DAY11-数据统计与内容审核

1、用户冻结解冻

用户冻结/解冻使用管理员在后台系统对用户的惩罚措施。对于发布不当言论或者违法违规内容的用户，可以暂时、永久禁止其登录，评论，发布动态、

后台中解冻/冻结，就是将用户状态写入数据库中

APP端用户在进行登录，评论，发布动态时检测Redis中冻结状态

接口文档如下 :

1.1 用户冻结

ManageController

tanhua-admin工程中ManageController编写冻结方法

//用户冻结
@PostMapping("/users/freeze")
public ResponseEntity freeze(@RequestBody Map params) {
    Map map =  managerService.userFreeze(params);
    return ResponseEntity.ok(map);
}

ManageService

tanhua-admin工程中ManageService编写冻结方法

    //用户冻结
    public Map userFreeze(Map params) {
        //1、构造key
        String userId = params.get("userId").toString();
        String key = Constants.USER_FREEZE + userId;
        //2、构造失效时间
        Integer freezingTime = Integer.valueOf(params.get("freezingTime").toString()); //冻结时间，1为冻结3天，2为冻结7天，3为永久冻结
        int days = 0;
        if(freezingTime == 1) {
            days = 3;
        }else if(freezingTime == 2) {
            days = 7;
        }
        //3、将数据存入redis
        String value = JSON.toJSONString(params);
        if(days>0) {
            redisTemplate.opsForValue().set(key,value,days, TimeUnit.MINUTES);
        }else {
            redisTemplate.opsForValue().set(key,value);
        }
        Map retMap = new HashMap();
        retMap.put("message","冻结成功");
        return retMap;
    }

1.2 用户解冻

tanhua-admin工程中ManageController编写解冻方法

ManageController

//用户解冻
@PostMapping("/users/unfreeze")
public ResponseEntity unfreeze(@RequestBody  Map params) {
    Map map =  managerService.userUnfreeze(params);
    return ResponseEntity.ok(map);
}

ManageService

tanhua-admin工程中ManageService编写解冻方法

//用户解冻
    public Map userUnfreeze(Map params) {
        String userId = params.get("userId").toString();
        String key = Constants.USER_FREEZE + userId;
        //删除redis数据
        redisTemplate.delete(key);
        Map retMap = new HashMap();
        retMap.put("message","解冻成功");
        return retMap;
    }

1.3 查询数据列表

UserInfo

在UserInfo实体类中添加字段，设置冻结状态：1-正常，2-已冻结

//用户状态,1为正常，2为冻结
@TableField(exist = false)
private String userStatus = "1";

ManageService

修改tanhua-admin工程中ManageService的根据id查询方法，设置冻结状态

    //根据id查询
    public UserInfo findUserById(Long userId) {
        UserInfo userInfo = userInfoApi.findById(userId);
        //查询redis中的冻结状态
        String key = Constants.USER_FREEZE + userId;
        if(redisTemplate.hasKey(key)) {
            userInfo.setUserStatus("2");
        }
        return userInfo;
    }

1.4 探花系统修改

在tanhua-app-server项目中新建UserFreezeService，编写校验方法。判断用户冻结状态

@Service
public class UserFreezeService {

    @Autowired
    private RedisTemplate<String,String> redisTemplate;

    /**
     * 判断用户是否被冻结，已被冻结，抛出异常
     *  参数：冻结范围，用户id
     *
     *  检测登录：
     *     checkUserStatus（“1”，106）
     */
    public void checkUserStatus(String state,Long userId) {
        //1、拼接key，从redis中查询数据
        String key = Constants.USER_FREEZE + userId;
        String value = redisTemplate.opsForValue().get(key);
        //2、如果数据存在，且冻结范围一致，抛出异常
        if(!StringUtils.isEmpty(value)) {
            Map map = JSON.parseObject(value, Map.class);
            String freezingRange = (String) map.get("freezingRange");
            if(state.equals(freezingRange)) {
                throw new BusinessException(ErrorResult.builder().errMessage("用户被冻结").build());
            }
        }
     }
}

tanhua-app-server项目发送短信验证码的接口, 需要校验用户是否被冻结

@Autowired
private UserFreezeService userFreezeService;

/**
 * 发送短信验证码
 *
 * @param phone
 */
public void sendCode(String phone) {
    //校验用户状态--------------------------
    User user = userApi.findByMobile(phone);
    if (user != null) {
        userFreezeService.checkUserStatus(1,user.getId());
    }

    //1. 生成随机验证码
    //String code = RandomStringUtils.randomNumeric(6);
    String code = "123456";
    //2. 调用阿里云发送验证码
    //判断验证码是否还未失效
    if (redisTemplate.hasKey("CHECK_CODE_" + phone)) {
        throw new RuntimeException("验证码还未失效");
    }
    //smsTemplate.sendSms(phone, code);
    //3. 将验证码保存到redis
    redisTemplate.opsForValue().set("CHECK_CODE_" + phone, code, Duration.ofMinutes(5));
}

tanhua-app-server模块中，发布动态和发布评论等接口也需要做相应的校验工作。

2、数据统计

后台系统首页中，显示各种统计数据，比如：累计用户数、新增用户数、登录次数等内容。

1、探花系统将用户操作日志写入RabbitMQ

2、管理后台获取最新消息，构造日志数据存入数据库

3、加入统计表，定时统计

2.1 数据采集

1、探花系统将用户操作日志写入RabbitMQ

2、管理后台获取最新消息，构造日志数据存入数据库

2.1.1 部署RabbitMQ

探花交友所需的第三方服务组件，已经以Docker-Compose准备好了。仅仅需要进入相关目录，以命令形式启动运行即可

#进入目录
cd /root/docker-file/rmq/
#创建容器并启动
docker-compose up –d
#查看容器
docker ps -a

服务地址：192.168.136.160:5672

管理后台：http://192.168.136.160:15672/ guest guest

2.1.2 消息类型说明

探花项目间使用RabbitMQ收发消息，这里采用topic类型消息

日志消息key规则：log.xxxx

2.1.3 实体类对象

Log

package com.tanhua.model.domain;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class Log {
    /**
     * id
     */
    private Long id;
    /**
     * 用户id
     */
    private Long userId;
    /**
     * 操作时间
     */
    private String logTime;

    /**
     * 操作类型,
     * 0101为登录，0102为注册，
     * 0201为发动态，0202为浏览动态，0203为动态点赞，0204为动态喜欢，0205为评论，0206为动态取消点赞，0207为动态取消喜欢，
     * 0301为发小视频，0302为小视频点赞，0303为小视频取消点赞，0304为小视频评论
     */
    private String type;

    /**
     * 登陆地点
     */
    private String place;
    /**
     * 登陆设备
     */
    private String equipment;

    public Log(Long userId, String logTime, String type) {
        this.userId = userId;
        this.logTime = logTime;
        this.type = type;
    }
}

Analysis

package com.tanhua.model.admin;

import com.baomidou.mybatisplus.annotation.TableName;
import com.tanhua.model.domain.BasePojo;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.Date;

@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder
@TableName("tb_analysis_by_day")
public class Analysis extends BasePojo {

    private Long id;
    /**
     * 日期
     */
    private Date recordDate;
    /**
     * 新注册用户数
     */
    private Integer numRegistered = 0;
    /**
     * 活跃用户数
     */
    private Integer numActive = 0;
    /**
     * 登陆次数
     */
    private Integer numLogin = 0;
    /**
     * 次日留存用户数
     */
    private Integer numRetention1d = 0;

    private Date created;

    private Date updated;
}

2.1.4 消息发送工具类

在tanhua-app-server模块中配置发送消息的工具类

package com.tanhua.server.service;

import com.alibaba.fastjson.JSON;
import org.springframework.amqp.AmqpException;
import org.springframework.amqp.core.AmqpTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;

@Service
public class MqMessageService {

    @Autowired
    private AmqpTemplate amqpTemplate;

    //发送日志消息
    public void sendLogMessage(Long userId,String type,String key,String busId) {
        try {
            Map map = new HashMap();
            map.put("userId",userId.toString());
            map.put("type",type);
            map.put("logTime",new SimpleDateFormat("yyyy-MM-dd").format(new Date()));
            map.put("busId",busId);
            String message = JSON.toJSONString(map);
            amqpTemplate.convertAndSend("tanhua.log.exchange", "log."+key,message);
        } catch (AmqpException e) {
            e.printStackTrace();
        }
    }

    //发送动态审核消息
    public void sendAudiMessage(String movementId) {
        try {
            amqpTemplate.convertAndSend("tanhua.audit.exchange",  "audit.movement",movementId);
        } catch (AmqpException e) {
            e.printStackTrace();
        }
    }
}

2.1.5 发送日志消息

/**
 * 发送日志消息
 *
 * @param userId
 * @param type 操作类型,
 *            0101为登录，
 *            0102为注册，
 *            0201为发动态，
 *            0202为浏览动态，
 *            0203为动态点赞，
 *            0204为动态喜欢，
 *            0205为评论，
 *            0206为动态取消点赞，
 *            0207为动态取消喜欢，
 *            0301为发小视频，
 *            0302为小视频点赞，
 *            0303为小视频取消点赞，
 *            0304为小视频评论
 * @param key      用户相关user , 动态相关movement , 小视频相关 video
 * @param busId   业务id  动态id或者视频id
 */
public void sendLogMessage(Long userId, String type, String key, String busId) 

在需要发送消息的位置, 调用上述方法传入指定参数发送消息即可

2.1.6 监听器处理消息

在tanhua-admin模块中配置消息监听器，监听日志消息。

package com.tanhua.admin.listener;

import com.alibaba.fastjson.JSON;
import com.tanhua.admin.mapper.LogMapper;
import com.tanhua.model.admin.Log;
import org.springframework.amqp.core.ExchangeTypes;
import org.springframework.amqp.rabbit.annotation.Exchange;
import org.springframework.amqp.rabbit.annotation.Queue;
import org.springframework.amqp.rabbit.annotation.QueueBinding;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.Map;

@Component
public class LogListener {

    @Autowired
    private LogMapper logMapper;

    @RabbitListener(bindings = @QueueBinding(
            value = @Queue(value = "tanhua.log.queue", durable = "true"),
            exchange = @Exchange(value = "tanhua.log.exchange", type = ExchangeTypes.TOPIC),
            key = {"log.*"}
        )
    )
    public void listenCreate(String message) throws Exception {
        try {
            Map<String, Object> map = JSON.parseObject(message);
            //1、获取数据
            Long userId = (Long) map.get("userId");
            String date = (String) map.get("date");
            String objId = (String) map.get("objId");
            String type = (String) map.get("type");
            //2、保存到数据库
            Log log = new Log(userId, date, type);
            logMapper.insert(log);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

2.2 定时任务

在实际项目开发中，除了Web应用、SOA服务外，还有一类不可缺少的，那就是定时任务调度。定时任务的场景可以说非常广泛：

• 某些网站会定时发送优惠邮件；

• 银行系统还款日信用卡催收款；

• 某些应用的生日祝福短信等。

那究竟何为定时任务调度，一句话概括就是：基于给定的时间点、给定的时间间隔、自动执行的任务

2.3.1 入门案例

开启定时任务

修改tanhua-admin模块引导类，开启SpringTask功能支持

@SpringBootApplication(exclude = {MongoAutoConfiguration.class, MongoDataAutoConfiguration.class})
@MapperScan("com.tanhua.admin.mapper")
@EnableScheduling //开启定时任务支持
public class AdminServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(AdminServerApplication.class,args);
    }
}

定时任务类

在tanhua-admin模块配置定时任务类

@Component
public class AnalysisTask {

    /**
     * 配置时间规则
     */
    @Scheduled( cron = "0/20 * * * * ? ")
    public void analysis() throws ParseException {
        //业务逻辑
        String time = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date());
        System.out.println("当前时间："+time);
    }

}

2.3.2 CRON表达式

对于定时任务，我们使用的时候主要是注重两个方面，一个是定时任务的业务，另一个就是Cron表达式。

*Cron 表达式支持到六个域 *

月份和星期的名称是不区分大小写的。FRI 和 fri 是一样的。

2.3.3 定时统计

AnalysisTask

修改AnalysisTask类，调用service进行定时统计

@Component
public class AnalysisTask {

    @Autowired
    private AnalysisService analysisService;

    /**
     * 配置时间规则
     *   在学习测试时，可以将时间间隔设置相对短一些
     */
    @Scheduled( cron = "0/20 * * * * ? ")
    public void analysis() throws ParseException {
        //业务逻辑
        String time = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date());
        System.out.println("开始统计："+time);
        //调logService完成日志统计
        analysisService.analysis();
        System.out.println("结束统计");
    }
}

AnalysisService

在tanhua-admin模块配置AnalysisService

package com.tanhua.admin.service;

import cn.hutool.core.date.DateUtil;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
import com.baomidou.mybatisplus.core.toolkit.Wrappers;
import com.tanhua.admin.mapper.AnalysisMapper;
import com.tanhua.admin.mapper.LogMapper;
import com.tanhua.model.admin.Analysis;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.expression.ParseException;
import org.springframework.stereotype.Service;

import java.text.SimpleDateFormat;
import java.util.Date;

/**
 * @author Administrator
 */
@Service
public class AnalysisService {

    @Autowired
    private AnalysisMapper analysisMapper;

    @Autowired
    private LogMapper logMapper;

       /**
     * 定时统计日志数据到统计表中
     * 1、查询tb_log表中的数 （每日注册用户数，每日登陆用户，活跃的用户数据，次日留存的用户）
     * 2、构造AnalysisByDay对象
     * 3、完成统计数据的更新或者保存
     */
      public void analysis() throws ParseException {
        //1、定义查询的日期
        String todayStr = new SimpleDateFormat("yyyy-MM-dd").format(new Date());
        String yesdayStr = DateUtil.yesterday().toString("yyyy-MM-dd");
        //2、统计数据-注册数量
        Integer regCount = logMapper.queryByTypeAndLogTime("0102", todayStr);
        //3、统计数据-登录数量
        Integer loginCount = logMapper.queryByTypeAndLogTime("0101", todayStr);
        //4、统计数据-活跃数量
        Integer activeCount = logMapper.queryByLogTime(todayStr);
        //5、统计数据-次日留存
        Integer numRetention1d = logMapper.queryNumRetention1d(todayStr, yesdayStr);
        //6、根据日期查询数据
        QueryWrapper<Analysis> qw = new QueryWrapper<Analysis>();
        qw.eq("record_date",new SimpleDateFormat("yyyy-MM-dd").parse(todayStr));
        //7、构造Analysis对象
        Analysis analysis = analysisMapper.selectOne(qw);
        //8、如果存在，更新，如果不存在保存
        if(analysis != null) {
            analysis.setNumRegistered(regCount);
            analysis.setNumLogin(loginCount);
            analysis.setNumActive(activeCount);
            analysis.setNumRetention1d(numRetention1d);
            analysisMapper.updateById(analysis);
        }else {
            analysis = new Analysis();
            analysis.setNumRegistered(regCount);
            analysis.setNumLogin(loginCount);
            analysis.setNumActive(activeCount);
            analysis.setNumRetention1d(numRetention1d);
            analysis.setRecordDate(new SimpleDateFormat("yyyy-MM-dd").parse(todayStr));
            analysis.setCreated(new Date());
            analysisMapper.insert(analysis);
        }
    }
}

LogMapper

在tanhua-admin模块中创建LogMapper并配置查询方法

package com.tanhua.admin.mapper;

import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.tanhua.model.admin.Log;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select;
import org.springframework.stereotype.Repository;

@Repository
public interface LogMapper extends BaseMapper<Log> {

    /**
     * 根据操作时间和类型统计日志统计用户数量
     *
     * @param type
     * @param logTime
     * @return
     */
    @Select("SELECT COUNT(DISTINCT user_id) FROM tb_log WHERE TYPE=#{type} AND log_time=#{logTime}")
    Integer queryByTypeAndLogTime(@Param("type") String type, @Param("logTime") String logTime);

    /**
     * 根据时间统计用户数量
     *
     * @param logTime
     * @return
     */
    @Select("SELECT COUNT(DISTINCT user_id) FROM tb_log WHERE log_time=#{logTime}")
    Integer queryByLogTime(String logTime);

    /**
     * 查询次日留存 , 从昨天活跃的用户中查询今日活跃用户
     *
     * @param today
     * @param yestoday
     * @return
     */
    @Select("SELECT COUNT(DISTINCT user_id)  FROM tb_log WHERE log_time=#{today} AND user_id IN (SELECT user_id FROM tb_log WHERE TYPE='0102' AND log_time=#{yestoday})")
    Integer queryNumRetention1d(@Param("today") String today, @Param("yestoday") String yestoday);
}

测试数据

为了方便操作，可以通过以下单元测试方法。保存若干操作数据

package com.tanhua.manager.test;

import com.tanhua.manager.domain.Log;
import com.tanhua.manager.mapper.LogMapper;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

import java.util.Random;

@RunWith(SpringRunner.class)
@SpringBootTest
public class LogTest {

    @Autowired
    private LogMapper logMapper;

    private String logTime = "";

    //模拟登录数据
    public void testInsertLoginLog() {
        for (int i = 0; i < 5; i++) {
            Log log = new Log();
            log.setUserId((long)(i+1));
            log.setLogTime(logTime);
            log.setType("0101");
            logMapper.insert(log);
        }
    }

    //模拟注册数据
    public void testInsertRegistLog() {
        for (int i = 0; i < 10; i++) {
            Log log = new Log();
            log.setUserId((long)(i+1));
            log.setLogTime(logTime);
            log.setType("0102");
            logMapper.insert(log);
        }
    }
    //模拟其他操作
    public void testInsertOtherLog() {
        String[] types = new String[]{"0201","0202","0203","0204","0205","0206","0207","0301","0302","0303","0304"};
        for (int i = 0; i < 10; i++) {
            Log log = new Log();
            log.setUserId((long)(i+1));
            log.setLogTime(logTime);
            int index = new Random().nextInt(10);
            log.setType(types[index]);
            logMapper.insert(log);
        }
    }

    @Test
    public void generData() {
        testInsertLoginLog();
        testInsertRegistLog();
        testInsertOtherLog();
    }
}

2.4 首页统计

2.4.1 vo对象

@Data
@AllArgsConstructor
@NoArgsConstructor
public class AnalysisSummaryVo {
    /**
     * 累计用户数
     */
    private Long cumulativeUsers;
    /**
     * 过去30天活跃用户数
     */
    private Long activePassMonth;
    /**
     * 过去7天活跃用户
     */
    private Long activePassWeek;
    /**
     * 今日新增用户数量
     */
    private Long newUsersToday;
    /**
     * 今日新增用户涨跌率，单位百分数，正数为涨，负数为跌
     */
    private BigDecimal newUsersTodayRate;
    /**
     * 今日登录次数
     */
    private Long loginTimesToday;
    /**
     * 今日登录次数涨跌率，单位百分数，正数为涨，负数为跌
     */
    private BigDecimal loginTimesTodayRate;
    /**
     * 今日活跃用户数量
     */
    private Long activeUsersToday;
    /**
     * 今日活跃用户涨跌率，单位百分数，正数为涨，负数为跌
     */
    private BigDecimal activeUsersTodayRate;
}

2.4.2 DashboardController

package com.tanhua.admin.controller;

import cn.hutool.core.date.DateTime;
import cn.hutool.core.date.DateUtil;
import com.tanhua.admin.service.AnalysisService;
import com.tanhua.model.admin.Analysis;
import com.tanhua.model.vo.admin.AnalysisSummaryVo;
import org.apache.commons.lang3.time.DateUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.math.BigDecimal;
import java.util.Calendar;
import java.util.Date;

@RestController
@RequestMapping("/dashboard")
public class DashboardController {

    @Autowired
    private AnalysisService analysisService;

    /**
     * 概要统计信息
     */
    @GetMapping("/summary")
    public AnalysisSummaryVo getSummary() {

        AnalysisSummaryVo analysisSummaryVo = new AnalysisSummaryVo();

        Date now = Calendar.getInstance().getTime();

        //累计用户数
        Integer total = analysisService.queryCumulativeUsers();
        analysisSummaryVo.setCumulativeUsers(Long.valueOf(total));

        //查询今日统计信息
        Analysis today_analysis = analysisService.querySummary(now);

        //过去30天活跃用户
        analysisSummaryVo.setActivePassMonth(Long.valueOf(today_analysis.getNumActive30()));

        //过去7天活跃用户
        analysisSummaryVo.setActivePassWeek(Long.valueOf(today_analysis.getNumActive7()));

        //今日活跃用户
        analysisSummaryVo.setActiveUsersToday(Long.valueOf(today_analysis.getNumActive()));

        //今日新增用户
        analysisSummaryVo.setNewUsersToday(Long.valueOf(today_analysis.getNumRegistered()));

        //今日新增用户涨跌率，单位百分数，正数为涨，负数为跌
        //查询昨日统计信息
        Analysis yes_analysis = analysisService.querySummary(DateUtils.addDays(now, -1));

        analysisSummaryVo.setNewUsersTodayRate(computeRate(Long.valueOf(today_analysis.getNumRegistered()), Long.valueOf(yes_analysis.getNumRegistered())));

        //今日登录次数
        analysisSummaryVo.setLoginTimesToday(Long.valueOf(today_analysis.getNumLogin()));

        //今日登录次数涨跌率，单位百分数，正数为涨，负数为跌
        analysisSummaryVo.setLoginTimesTodayRate(computeRate(Long.valueOf(today_analysis.getNumLogin()), Long.valueOf(yes_analysis.getNumLogin())));

        //活跃用户涨跌率，单位百分数，正数为涨，负数为跌
        analysisSummaryVo.setActiveUsersTodayRate(computeRate(Long.valueOf(today_analysis.getNumActive()), Long.valueOf(yes_analysis.getNumActive())));

        return analysisSummaryVo;
    }

    private static BigDecimal computeRate(Long current, Long last) {
        BigDecimal result;
        if (last == 0) {
            // 当上一期计数为零时，此时环比增长为倍数增长
            result = new BigDecimal((current - last) * 100);
        } else {
            result = BigDecimal.valueOf((current - last) * 100).divide(BigDecimal.valueOf(last), 2, BigDecimal.ROUND_HALF_DOWN);
        }
        return result;
    }

    private static String offsetDay(Date date, int offSet) {
        return DateUtil.offsetDay(date, offSet).toDateStr();
    }
}

2.4.3 AnalysisService

public Integer queryCumulativeUsers() {
    return analysisMapper.queryCumulativeUsers();
}

public Analysis querySummary(Date now) {
    //5、根据当前时间查询AnalysisByDay数据
    LambdaQueryWrapper<Analysis> qw = Wrappers.<Analysis>lambdaQuery();
    qw.eq(Analysis::getRecordDate, DateUtil.format(now, "yyyy-MM-dd"));
    return analysisMapper.selectOne(qw);
}

2.4.4 AnalysisMapper

package com.tanhua.admin.mapper;

import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.tanhua.model.admin.Analysis;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select;
import org.springframework.stereotype.Repository;

/**
 * @author Administrator
 */
@Repository
public interface AnalysisMapper extends BaseMapper<Analysis> {

    @Select("select sum(num_registered) from tb_analysis")
    Integer queryCumulativeUsers();
}

3、内容审核

内容安全是识别服务，支持对图片、视频、文本、语音等对象进行多样化场景检测，有效降低内容违规风险。

目前很多平台都支持内容检测，如阿里云、腾讯云、百度AI、网易云等国内大型互联网公司都对外提供了API。

按照性能和收费来看，探花交友项目使用的就是阿里云的内容安全接口，使用到了图片和文本的审核。

3.1 阿里云内容审核

3.1.1 准备工作

1，前往阿里云官网注册账号

2，打开云盾内容安全产品试用页面，单击立即开通，正式开通服务

3，在AccessKey管理页面管理您的AccessKeyID和AccessKeySecret

3.1.2 文本内容垃圾检测

文本垃圾内容检测：点击访问

文本垃圾内容Java SDK: 点击访问

3.1.3 图片审核

图片垃圾内容Java SDK: https://help.aliyun.com/document_detail/53424.html?spm=a2c4g.11186623.6.715.c8f69b12ey35j4

3.1.4 抽取工具

GreenProperties

@Data
@ConfigurationProperties("tanhua.green")
public class GreenProperties {
    /**
     * 账号
     */
    String accessKeyID;
    /**
     * 密钥
     */
    String accessKeySecret;

    /**
     * 场景
     */
    String scenes;
}

AliyunGreenTemplate

package com.tanhua.autoconfig.template;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.green.model.v20180509.ImageSyncScanRequest;
import com.aliyuncs.green.model.v20180509.TextScanRequest;
import com.aliyuncs.http.FormatType;
import com.aliyuncs.http.HttpResponse;
import com.aliyuncs.http.MethodType;
import com.aliyuncs.http.ProtocolType;
import com.aliyuncs.profile.DefaultProfile;
import com.aliyuncs.profile.IClientProfile;
import com.tanhua.autoconfig.properties.GreenProperties;
import lombok.extern.slf4j.Slf4j;

import java.util.*;

/**
 * @author: itheima
 * @create: 2021-05-31 00:46
 */
@Slf4j
public class AliyunGreenTemplate {

    private IAcsClient client;

    private GreenProperties greenProperties;

    public AliyunGreenTemplate(GreenProperties greenProperties) {
        this.greenProperties = greenProperties;
        try {
            IClientProfile profile = DefaultProfile
                    .getProfile("cn-shanghai", greenProperties.getAccessKeyID(), greenProperties.getAccessKeySecret());
            DefaultProfile
                    .addEndpoint("cn-shanghai", "cn-shanghai", "Green", "green.cn-shanghai.aliyuncs.com");
            client = new DefaultAcsClient(profile);
        } catch (Exception e) {
            e.printStackTrace();
            log.error("Green配置缺失，请补充!");
        }
    }


    /**
     * 阿里云文本内容检查
     *
     * @param content
     * @return map  key - suggestion内容
     * pass：文本正常，可以直接放行，
     * review：文本需要进一步人工审核，
     * block：文本违规，可以直接删除或者限制公开
     * value -   通过，或 出错原因
     * @throws Exception
     */
    public Map<String, String> greenTextScan(String content) throws Exception {
        TextScanRequest textScanRequest = new TextScanRequest();
        textScanRequest.setAcceptFormat(FormatType.JSON); // 指定api返回格式
        textScanRequest.setHttpContentType(FormatType.JSON);
        textScanRequest.setMethod(MethodType.POST); // 指定请求方法
        textScanRequest.setEncoding("UTF-8");
        textScanRequest.setRegionId("cn-shanghai");
        List<Map<String, Object>> tasks = new ArrayList<>();
        Map<String, Object> task1 = new LinkedHashMap<>();
        task1.put("dataId", UUID.randomUUID().toString());
        /**
         * 待检测的文本，长度不超过10000个字符
         */
        task1.put("content", content);
        tasks.add(task1);
        JSONObject data = new JSONObject();

        /**
         * 检测场景，文本垃圾检测传递：antispam
         **/
        data.put("scenes", Arrays.asList("antispam"));
        data.put("tasks", tasks);
        log.info("检测任务内容：{}", JSON.toJSONString(data, true));
        textScanRequest.setHttpContent(data.toJSONString().getBytes("UTF-8"), "UTF-8", FormatType.JSON);
        // 请务必设置超时时间
        textScanRequest.setConnectTimeout(3000);
        textScanRequest.setReadTimeout(6000);

//        返回结果内容
        Map<String, String> resultMap = new HashMap<>();
        try {
            HttpResponse httpResponse = client.doAction(textScanRequest);
            if (!httpResponse.isSuccess()) {
                new RuntimeException("阿里云文本内容检查出现异常！");
            }
            JSONObject scrResponse = JSON.parseObject(new String(httpResponse.getHttpContent(), "UTF-8"));
            log.info("检测结果内容：{}", JSON.toJSONString(scrResponse, true));
            if (200 != scrResponse.getInteger("code")) {
                new RuntimeException("阿里云文本内容检查出现异常！");
            }
            JSONArray taskResults = scrResponse.getJSONArray("data");
            for (Object taskResult : taskResults) {
                if (200 != ((JSONObject) taskResult).getInteger("code")) {
                    new RuntimeException("阿里云文本内容检查出现异常！");
                }
                JSONArray sceneResults = ((JSONObject) taskResult).getJSONArray("results");
                for (Object sceneResult : sceneResults) {
                    String scene = ((JSONObject) sceneResult).getString("scene");
                    String label = ((JSONObject) sceneResult).getString("label");
                    String suggestion = ((JSONObject) sceneResult).getString("suggestion");
                    log.info("最终内容检测结果，suggestion = {}，label={}", suggestion, label);
//                    设置默认错误返回内容
                    resultMap.put("suggestion", suggestion);
                    if (suggestion.equals("review")) {
                        resultMap.put("reson", "文章内容中有不确定词汇");
                        log.info("返回结果，resultMap={}", resultMap);
                        return resultMap;
                    } else if (suggestion.equals("block")) {
                        String reson = "文章内容中有敏感词汇";
                        if (label.equals("spam")) {
                            reson = "文章内容中含垃圾信息";
                        } else if (label.equals("ad")) {
                            reson = "文章内容中含有广告";
                        } else if (label.equals("politics")) {
                            reson = "文章内容中含有涉政";
                        } else if (label.equals("terrorism")) {
                            reson = "文章内容中含有暴恐";
                        } else if (label.equals("abuse")) {
                            reson = "文章内容中含有辱骂";
                        } else if (label.equals("porn")) {
                            reson = "文章内容中含有色情";
                        } else if (label.equals("flood")) {
                            reson = "文章内容灌水";
                        } else if (label.equals("contraband")) {
                            reson = "文章内容违禁";
                        } else if (label.equals("meaningless")) {
                            reson = "文章内容无意义";
                        }
                        resultMap.put("reson", reson);
                        log.info("返回结果，resultMap={}", resultMap);
                        return resultMap;
                    }

                }
            }
            resultMap.put("suggestion", "pass");
            resultMap.put("reson", "检测通过");

        } catch (Exception e) {
            log.error("阿里云文本内容检查出错！");
            e.printStackTrace();
            new RuntimeException("阿里云文本内容检查出错！");
        }
        log.info("返回结果，resultMap={}", resultMap);
        return resultMap;
    }

    /**
     * 阿里云图片内容安全
     */
    public Map imageScan(List<String> imageList) throws Exception {
        IClientProfile profile = DefaultProfile
                .getProfile("cn-shanghai", greenProperties.getAccessKeyID(), greenProperties.getAccessKeySecret());
        ImageSyncScanRequest imageSyncScanRequest = new ImageSyncScanRequest();
        // 指定api返回格式
        imageSyncScanRequest.setAcceptFormat(FormatType.JSON);
        // 指定请求方法
        imageSyncScanRequest.setMethod(MethodType.POST);
        imageSyncScanRequest.setEncoding("utf-8");
        //支持http和https
        imageSyncScanRequest.setProtocol(ProtocolType.HTTP);
        JSONObject httpBody = new JSONObject();
        /**
         * 设置要检测的场景, 计费是按照该处传递的场景进行
         * 一次请求中可以同时检测多张图片，每张图片可以同时检测多个风险场景，计费按照场景计算
         * 例如：检测2张图片，场景传递porn、terrorism，计费会按照2张图片鉴黄，2张图片暴恐检测计算
         * porn: porn表示色情场景检测
         */

        httpBody.put("scenes", Arrays.asList(greenProperties.getScenes().split(",")));

        /**
         * 如果您要检测的文件存于本地服务器上，可以通过下述代码片生成url
         * 再将返回的url作为图片地址传递到服务端进行检测
         */
        /**
         * 设置待检测图片， 一张图片一个task
         * 多张图片同时检测时，处理的时间由最后一个处理完的图片决定
         * 通常情况下批量检测的平均rt比单张检测的要长, 一次批量提交的图片数越多，rt被拉长的概率越高
         * 这里以单张图片检测作为示例, 如果是批量图片检测，请自行构建多个task
         */
        List list = new ArrayList();
        for (String imageUrl : imageList) {
            JSONObject task = new JSONObject();
            task.put("dataId", UUID.randomUUID().toString());
            // 设置图片链接。
            task.put("url", imageUrl);
            task.put("time", new Date());
            list.add(task);
        }

        httpBody.put("tasks",list);

        imageSyncScanRequest.setHttpContent(org.apache.commons.codec.binary.StringUtils.getBytesUtf8(httpBody.toJSONString()),
                "UTF-8", FormatType.JSON);
        /**
         * 请设置超时时间, 服务端全链路处理超时时间为10秒，请做相应设置
         * 如果您设置的ReadTimeout小于服务端处理的时间，程序中会获得一个read timeout异常
         */
        imageSyncScanRequest.setConnectTimeout(3000);
        imageSyncScanRequest.setReadTimeout(10000);
        HttpResponse httpResponse = null;
        try {
            httpResponse = client.doAction(imageSyncScanRequest);
        } catch (Exception e) {
            e.printStackTrace();
        }

        Map<String, String> resultMap = new HashMap<>();

        //服务端接收到请求，并完成处理返回的结果
        if (httpResponse != null && httpResponse.isSuccess()) {
            JSONObject scrResponse = JSON.parseObject(org.apache.commons.codec.binary.StringUtils.newStringUtf8(httpResponse.getHttpContent()));
            System.out.println(JSON.toJSONString(scrResponse, true));
            int requestCode = scrResponse.getIntValue("code");
            //每一张图片的检测结果
            JSONArray taskResults = scrResponse.getJSONArray("data");
            if (200 == requestCode) {
                for (Object taskResult : taskResults) {
                    //单张图片的处理结果
                    int taskCode = ((JSONObject) taskResult).getIntValue("code");
                    //图片要检测的场景的处理结果, 如果是多个场景，则会有每个场景的结果
                    JSONArray sceneResults = ((JSONObject) taskResult).getJSONArray("results");
                    if (200 == taskCode) {
                        for (Object sceneResult : sceneResults) {
                            String scene = ((JSONObject) sceneResult).getString("scene");
                            String label = ((JSONObject) sceneResult).getString("label");
                            String suggestion = ((JSONObject) sceneResult).getString("suggestion");
                            //根据scene和suggetion做相关处理
                            //do something
                            System.out.println("scene = [" + scene + "]");
                            System.out.println("suggestion = [" + suggestion + "]");
                            System.out.println("suggestion = [" + label + "]");
                            if (!suggestion.equals("pass")) {
                                resultMap.put("suggestion", suggestion);
                                resultMap.put("label", label);
                                return resultMap;
                            }
                        }

                    } else {
                        //单张图片处理失败, 原因视具体的情况详细分析
                        log.error("task process fail. task response:" + JSON.toJSONString(taskResult));
                        return null;
                    }
                }
                resultMap.put("suggestion", "pass");
                return resultMap;
            } else {
                /**
                 * 表明请求整体处理失败，原因视具体的情况详细分析
                 */
                log.error("the whole image scan request failed. response:" + JSON.toJSONString(scrResponse));
                return null;
            }
        }
        return null;
    }
}

TanhuaAutoConfiguration

@Bean
@ConditionalOnProperty(prefix = "tanhua.green",value = "enable", havingValue = "true")
public AliyunGreenTemplate aliyunGreenTemplate(GreenProperties properties) {
    return new AliyunGreenTemplate(properties);
}

配置文件

tanhua:
  green:
    enable: true
    accessKeyID: LTAI4GKgob9vZ53k2SZdyAC7
    accessKeySecret: LHLBvXmILRoyw0niRSBuXBZewQ30la
    scenes: porn,terrorism #色情，暴力

单元测试

package com.tanhua.admin;

import com.tanhua.autoconfig.template.AliyunGreenTemplate;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

@RunWith(SpringRunner.class)
@SpringBootTest
public class GreenTest {

    @Autowired
    private AliyunGreenTemplate template;

    @Test
    public void test() throws Exception {
//        Long data = analysisMapper.sumAnalysisData("num_registered", "2020-09-14", "2020-09-18");
//        System.out.println(data);
//        Map<String, String> map = template.greenTextScan("本校小额贷款，安全、快捷、方便、无抵押，随机随贷，当天放款，上门服务");
//        map.forEach((k,v)-> System.out.println(k +"--" + v));
        List<String> list = new ArrayList<>();
       list.add("http://p7.itc.cn/images01/20210707/ecdf3bf5aaf34c67b305d3d52e0d72af.jpeg");
        Map<String, String> map = template.imageScan(list);
        System.out.println("------------");
        map.forEach((k, v) -> System.out.println(k + "--" + v));
    }
}

3.2 动态审核

3.2.1 执行流程

为了解决程序间耦合关系，这里采用RabbitMQ + 阿里云完成内容审核

• 用户发布动态，保存到数据库

• 发送RabbitMQ消息

• 管理后台监听消息，对内容（文本、图片审核）

• 更新动态的状态

3.2.2 发布消息

修改tanhua-app-server模块中发布动态的方法，当动态发布完成，发送一条MQ消息

/**
 * 发布动态
 *
 * @param movement
 * @param imageContent
 */
public void publish(Movement movement, MultipartFile[] imageContent) throws IOException {
    //1. 参数校验
    if (movement == null || (StringUtils.isEmpty(movement.getTextContent()) && ArrayUtil.isEmpty(imageContent))) {
        throw new BusinessException(ErrorResult.contentError());
    }
    //2. 上传图片到阿里云
    List<String> medias = new ArrayList<>();
    if (ArrayUtil.isNotEmpty(imageContent)) {
        for (MultipartFile file : imageContent) {
            String url = ossTemplate.upload(file.getOriginalFilename(), file.getInputStream());
            medias.add(url);
        }
    }
    //3. 调用dubbo服务发布动态
    Long userId = UserHolder.getUserId();
    movement.setUserId(userId);
    movement.setMedias(medias);
    movement.setState(0);

    movementApi.publish(movement);

    //发送动态审核消息
    amqpTemplate.convertSendAndReceive("tanhua.audit.exchange", "audit.movement", movement.getId().toHexString());
}

3.2.3 监听器

在tanhua-admin模块中配置内容审核的监听器

package com.tanhua.admin.listener;

import com.tanhua.autoconfig.template.AliyunGreenTemplate;
import com.tanhua.dubbo.api.mongo.MovementApi;
import com.tanhua.model.mongo.Movement;
import org.apache.dubbo.config.annotation.DubboReference;
import org.bson.types.ObjectId;
import org.springframework.amqp.core.ExchangeTypes;
import org.springframework.amqp.rabbit.annotation.Exchange;
import org.springframework.amqp.rabbit.annotation.Queue;
import org.springframework.amqp.rabbit.annotation.QueueBinding;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.Map;

@Component
public class MovementListener {

    @DubboReference
    private MovementApi movementApi;

    @Autowired
    private AliyunGreenTemplate aliyunGreenTemplate;

    @RabbitListener(bindings = @QueueBinding(
            value = @Queue(value = "tanhua.audit.queue", durable = "true"),
            exchange = @Exchange(value = "tanhua.audit.exchange", type = ExchangeTypes.TOPIC),
            key = {"audit.movement"}
    )
    )
    public void listenCreate(String movementId) throws Exception {
        try {
            //1、根据动态id查询动态
            Movement movement = movementApi.findById(new ObjectId(movementId));
            //对于RocketMQ消息有可能出现重复，解决方法判断 (幂等性)
            Integer state = 0;
            if (movement != null && movement.getState() == 0) {
                Map<String, String> textScan = aliyunGreenTemplate.greenTextScan(movement.getTextContent());
                Map<String, String> imageScan = aliyunGreenTemplate.imageScan(movement.getMedias());
                if (textScan != null && imageScan != null) {
                    String textSuggestion = textScan.get("suggestion");
                    String imageSuggestion = imageScan.get("suggestion");
                    if ("block".equals(textSuggestion) || "block".equals(textSuggestion)) {
                        state = 2;
                    } else if ("pass".equals(textSuggestion) || "pass".equals(textSuggestion)) {
                        state = 1;
                    }
                }
            }
            movementApi.update(movementId, state);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

3.2.4 movementApi

修改tanhua-dubbo-mongo模块中的MovementApi与实现类，添加修改状态的方法

@Override
public void update(String movementId, Integer state) {
    Query query = Query.query(Criteria.where("id").in(new ObjectId(movementId)));
    Update update = Update.update("state", state);
    mongoTemplate.updateFirst(query, update, Movement.class);
}