您现在的位置是：首页 >技术交流 >【精选】基于Spark的共享单车数据存储系统设计与实现（大数据组件+Spark+Hive+MySQL+AI智能云）数据存储、分析与预测系统 共享单车数据存储与分析系统网站首页技术交流
【精选】基于Spark的共享单车数据存储系统设计与实现（大数据组件+Spark+Hive+MySQL+AI智能云）数据存储、分析与预测系统 共享单车数据存储与分析系统
简介【精选】基于Spark的共享单车数据存储系统设计与实现（大数据组件+Spark+Hive+MySQL+AI智能云）数据存储、分析与预测系统 共享单车数据存储与分析系统
博主介绍：
✌我是阿龙，一名专注于Java技术领域的程序员，全网拥有10W+粉丝。作为CSDN特邀作者、博客专家、新星计划导师，我在计算机毕业设计开发方面积累了丰富的经验。同时，我也是掘金、华为云、阿里云、InfoQ等平台的优质作者。通过长期分享和实战指导，我致力于帮助更多学生完成毕业项目和技术提升。技术范围：
我熟悉的技术领域涵盖SpringBoot、Vue、SSM、HLMT、Jsp、PHP、Nodejs、Python、爬虫、数据可视化、小程序、安卓app、大数据、物联网、机器学习等方面的设计与开发。如果你有任何技术难题，我都乐意与你分享解决方案。主要内容：
我的服务内容包括：免费功能设计、开题报告、任务书、中期检查PPT、系统功能实现、代码编写、论文撰写与辅导、论文降重、长期答辩答疑辅导。我还提供腾讯会议一对一的专业讲解和模拟答辩演练，帮助你全面掌握答辩技巧与代码逻辑。🍅获取源码请在文末联系我🍅
温馨提示：文末有 CSDN 平台官方提供的阿龙联系方式的名片！
温馨提示：文末有 CSDN 平台官方提供的阿龙联系方式的名片！
温馨提示：文末有 CSDN 平台官方提供的阿龙联系方式的名片！
感兴趣的可以先收藏起来，还有大家在毕设选题，项目以及论文编写等相关问题都可以给我留言咨询，希望帮助更多的人
目录：
感兴趣的可以先收藏起来，还有大家在毕设选题，项目以及论文编写等相关问题都可以给我留言咨询，希望帮助更多的人
源码获取文章下方名片联系我即可~大家点赞、收藏、关注、评论啦 、查看👇🏻获取联系方式👇🏻精彩专栏推荐订阅：在下方专
一、详细操作演示视频
在文章的尾声，您会发现一张电子名片👤，欢迎通过名片上的联系方式与我取得联系，以获取更多关于项目演示的详尽视频内容。视频将帮助您全面理解项目的关键点和操作流程。期待与您的进一步交流！
系统简介：
随着共享经济的兴起，共享单车作为一种新兴的交通方式迅速普及。它不仅解决了“最后一公里”的出行问题，还有助于缓解城市交通压力。然而，随着用户数量和单车规模的快速增长，如何高效地管理和存储海量的共享单车数据成为了一个挑战。这些数据包括用户的骑行记录、车辆的位置信息、使用状态等，对于保障服务的可靠性、提升用户体验以及进行城市交通规划都至关重要。因此，研发一个专门针对共享单车的数据存储系统显得尤为迫切。
共享单车数据存储系统的开发对于整个共享单车行业的健康发展具有深远的意义。首先，一个高效的数据存储系统能够确保数据的快速处理和安全存储，为运营商提供准确的数据支持，帮助其优化运营策略，提高服务质量。其次，通过对大量数据的分析，系统可以为城市交通规划提供有价值的参考，促进城市交通的合理布局。此外，良好的数据管理还能够提高用户满意度，增强用户对共享单车服务的信任和依赖。综上所述，共享单车数据存储系统的建立对于提升行业竞争力、推动智慧城市建设等方面都具有重要作用。
第2章 系统开发技术简介
2.1 Python语言
Python是一种高级编程语言，以其简洁易读的语法和强大的功能而闻名。Python的设计哲学强调代码的可读性和简洁性，这使得Python成为了许多开发者的首选语言。Python支持多种编程范式，包括面向对象编程、过程式编程和函数式编程。面向对象编程是Python的核心特性之一，它允许程序员通过创建类和对象来组织和管理代码。类是一种用于定义对象的蓝图，对象是类的实例。通过使用类和对象，Python提供了一种灵活的方式来表示现实世界中的事物。过程式编程是一种编程范式，它强调编写一系列简单的步骤来解决问题。Python支持过程式编程，这意味着你可以使用循环、条件语句和函数等基本结构来编写程序。这种编程范式使得Python非常适合于解决需要重复执行相同任务的问题。函数式编程是一种编程范式，它强调使用纯函数和不可变性来编写程序。纯函数是指没有副作用的函数，它们的结果只依赖于输入参数。不可变性是指一个对象的状态在创建后不能被修改。Python支持函数式编程，这意味着你可以使用高阶函数（如map、filter和reduce等）来处理数据集合。Python还具有丰富的标准库和第三方库，这些库提供了许多实用的功能，如文件操作、网络编程、数据库访问等。这些库使得Python成为了一个功能强大的开发工具，可以帮助你快速构建各种类型的应用程序。Python是一种强大且易于学习的编程语言，它具有简洁的语法、丰富的功能和广泛的应用领域。
2.2 Django框架
Django是一个基于Python的高级Web框架，它鼓励快速开发和干净、实用的设计。Django的核心理念是“简洁明了”，它提供了许多内置的功能，如URL路由、模板引擎、表单处理等，使得开发者可以专注于业务逻辑的开发，而无需关心底层的细节。Django的MVC（Model-View-Controller）架构模式使得开发者可以将应用程序分为三个部分：模型（Model）、视图（View）和控制器（Controller）。模型负责处理数据存储和检索，视图负责处理用户界面和业务逻辑，控制器负责协调模型和视图之间的交互。这种架构模式使得代码更加模块化和易于维护。Django还支持数据库抽象层，通过ORM（Object-Relational Mapping）技术，将数据库操作转换为面向对象的操作。这使得开发者可以使用Python语言编写数据库查询，而无需关心底层的SQL语句。此外，Django还支持多种数据库系统，如MySQL、PostgreSQL和SQLite等。Django还提供了丰富的第三方库和插件，这些库和插件可以帮助开发者快速实现各种功能，如用户认证、表单处理、文件上传等。这些库和插件通常由社区成员和维护，因此它们通常是稳定且安全的。Django是一个功能强大、易于学习的Web框架，它提供了许多内置的功能和第三方库，使得开发者可以快速构建高质量的Web应用程序。
2.3 Hadoop介绍
Hadoop是一个开源的分布式计算框架，它允许用户在大量的计算机集群上进行数据处理和分析。Hadoop的核心组件包括HDFS（Hadoop Distributed File System）和MapReduce。HDFS是一个高度容错的、可扩展的分布式文件系统，它可以将大量数据分布在多个节点上，提供高吞吐量的数据访问。MapReduce则是一种编程模型，用于处理和生成大数据集。
Hadoop的设计目标是能够在廉价硬件上可靠地存储和处理海量数据。它提供了一种简单而强大的方法来处理大规模数据集，使得数据分析变得更加容易和高效。Hadoop已经被广泛应用于各种领域，如互联网搜索、日志分析、机器学习等。
2.4 Scrapy介绍
Scrapy是一个用于爬取网站数据并提取结构性数据的Python库。它提供了一个强大的框架，可以处理各种类型的网站，包括静态和动态网站。Scrapy使用异步网络协议（如Twisted或Tornado）来处理请求和响应，从而大大提高了爬取速度。它还具有高度可扩展性和可定制性，可以轻松地添加新的功能和扩展。
Scrapy的主要组件包括Scrapy引擎、调度器、爬虫、管道和项目。Scrapy引擎负责控制数据流在系统的各个组件之间传输，调度器负责确定下一个要爬取的URL，爬虫负责从网页中提取数据，管道负责处理和清理爬取到的数据，项目则是整个爬虫系统的容器。
Scrapy支持多种数据存储方式，包括文件系统、数据库和搜索引擎等。它还提供了丰富的中间件和扩展，可以方便地与其他工具和技术集成。
总之，Scrapy是一个功能强大、灵活易用的Python爬虫框架，适用于各种规模的网络数据采集任务。
2.5 MySQL数据库简介
MySQL是一个开源的关系型数据库管理系统，它使用结构化查询语言（SQL）作为访问和管理数据的主要方式。MySQL最初由瑞典MySQL AB公司开发，后来被甲骨文公司收购。MySQL是最受欢迎的关系型数据库之一，广泛应用于各种规模的企业和组织中。MySQL具有高性能、稳定性和易用性等特点。它支持多线程处理，可以充分利用现代计算机的多核处理器，提高数据库的执行效率。MySQL还具有高可靠性和可扩展性，可以在大量并发用户访问的情况下保持稳定运行。MySQL提供了丰富的功能和工具，可以帮助开发人员快速构建复杂的应用程序。它支持事务处理、存储过程、触发器等高级功能，可以满足各种业务需求。MySQL还提供了多种客户端工具，如命令行客户端、图形界面客户端等，方便用户管理和操作数据库。在安装和使用MySQL时，用户可以根据自己的需求选择不同的发行版。目前市面上有多个版本的MySQL可供选择，包括社区版、企业版等。社区版是免费的，但功能相对较少；企业版则是收费的，但提供了更多的功能和服务。MySQL是一个功能强大、性能优越、易于使用的关系型数据库管理系统。它广泛应用于各种规模的企业和组织中，为用户提供了稳定可靠的数据存储和访问服务。
2.6 B/S结构
B/S结构，浏览器/服务器（Browser/Server）结构，是一种网络应用程序的体系结构。在这种结构中，客户端通过浏览器访问服务器上的应用程序，服务器负责处理客户端的请求并返回相应的数据。这种模式使得用户无需安装专门的客户端软件，只需使用通用的浏览器即可访问应用程序，大大降低了系统的维护成本和升级难度。
B/S结构的优点是显而易见的：它具有良好的跨平台性，因为浏览器是通用的；它简化了客户端的开发和维护工作，因为所有的功能都在服务器端实现；它具有较高的安全性，服务器端可以对数据进行统一管理和保护。
B/S结构也存在一些缺点。由于所有数据都存储在服务器端，当服务器出现故障时，整个系统可能会瘫痪；B/S结构的性能受到网络带宽的限制，尤其是在大量数据传输的情况下；B/S结构的可扩展性相对较差，随着用户数量的增加，服务器端的负载也会相应增加。
系统总体框架：
核心代码介绍:
# coding:utf-8
__author__ = "ila"
import logging, os, json, configparser
import time
from datetime import datetime
from flask import request, jsonify,session
from sqlalchemy.sql import func,and_,or_,case
from sqlalchemy import cast, Integer,Float
from api.models.brush_model import *
from . import main_bp
from utils.codes import *
from utils.jwt_auth import Auth
from configs import configs
from utils.helper import *
import random
import smtplib
from email.mime.text import MIMEText
from email.utils import formataddr
from email.header import Header
from utils.baidubce_api import BaiDuBce
from api.models.config_model import config
from flask import current_app as app
from utils.spark_func import spark_read_mysql
from utils.hdfs_func import upload_to_hdfs
from utils.mapreduce1 import MRMySQLAvg
# 注册接口
@main_bp.route("/python501552a0/gongxiangdanche/register", methods=['POST'])
def python501552a0_gongxiangdanche_register():
if request.method == 'POST':
msg = {'code': normal_code, 'message': 'success', 'data': [{}]}
req_dict = session.get("req_dict")
error = gongxiangdanche.createbyreq(gongxiangdanche, gongxiangdanche, req_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = "注册用户已存在"
return jsonify(msg)
# 登录接口
@main_bp.route("/python501552a0/gongxiangdanche/login", methods=['GET','POST'])
def python501552a0_gongxiangdanche_login():
if request.method == 'GET' or request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
req_model = session.get("req_dict")
try:
del req_model['role']
except:
pass
datas = gongxiangdanche.getbyparams(gongxiangdanche, gongxiangdanche, req_model)
if not datas:
msg['code'] = password_error_code
msg['msg']='密码错误或用户不存在'
return jsonify(msg)
req_dict['id'] = datas[0].get('id')
try:
del req_dict['mima']
except:
pass
return Auth.authenticate(Auth, gongxiangdanche, req_dict)
# 登出接口
@main_bp.route("/python501552a0/gongxiangdanche/logout", methods=['POST'])
def python501552a0_gongxiangdanche_logout():
if request.method == 'POST':
msg = {
"msg": "退出成功",
"code": 0
}
req_dict = session.get("req_dict")
return jsonify(msg)
# 重置密码接口
@main_bp.route("/python501552a0/gongxiangdanche/resetPass", methods=['POST'])
def python501552a0_gongxiangdanche_resetpass():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success"}
req_dict = session.get("req_dict")
if req_dict.get('mima') != None:
req_dict['mima'] = '123456'
error = gongxiangdanche.updatebyparams(gongxiangdanche, gongxiangdanche, req_dict)
if error != None:
msg['code'] = crud_error_code
msg['msg'] = error
else:
msg['msg'] = '密码已重置为：123456'
return jsonify(msg)
# 获取会话信息接口
@main_bp.route("/python501552a0/gongxiangdanche/session", methods=['GET'])
def python501552a0_gongxiangdanche_session():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "data": {}}
req_dict={"id":session.get('params').get("id")}
msg['data'] = gongxiangdanche.getbyparams(gongxiangdanche, gongxiangdanche, req_dict)[0]
return jsonify(msg)
# 分类接口（后端）
@main_bp.route("/python501552a0/gongxiangdanche/page", methods=['GET'])
def python501552a0_gongxiangdanche_page():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data":{"currPage":1,"totalPage":1,"total":1,"pageSize":10,"list":[]}}
req_dict = session.get("req_dict")
userinfo = session.get("params")
try:
__hasMessage__=gongxiangdanche.__hasMessage__
except:
__hasMessage__=None
if __hasMessage__ and __hasMessage__!="否":
tablename=session.get("tablename")
if tablename!="users" and session.get("params")!=None and gongxiangdanche!='chat':
req_dict["userid"]=session.get("params").get("id")
tablename=session.get("tablename")
if tablename=="users" :
try:
pass
except:
pass
else:
mapping_str_to_object = {}
for model in Base_model._decl_class_registry.values():
if hasattr(model, '__tablename__'):
mapping_str_to_object[model.__tablename__] = model
try:
__isAdmin__=mapping_str_to_object[tablename].__isAdmin__
except:
__isAdmin__=None
if __isAdmin__!="是" and session.get("params")!=None:
req_dict["userid"]=session.get("params").get("id")
else:
try:
del req_dict["userid"]
except:
pass
clause_args = []
or_clauses = or_(*clause_args)
msg['data']['list'], msg['data']['currPage'], msg['data']['totalPage'], msg['data']['total'],
msg['data']['pageSize'] = gongxiangdanche.page(gongxiangdanche, gongxiangdanche, req_dict, or_clauses)
return jsonify(msg)
# 排序接口
@main_bp.route("/python501552a0/gongxiangdanche/autoSort", methods=['GET'])
def python501552a0_gongxiangdanche_autosort():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data":{"currPage":1,"totalPage":1,"total":1,"pageSize":10,"list":[]}}
req_dict = session.get("req_dict")
req_dict['sort']='clicktime'
req_dict['order']='desc'
try:
__browseClick__= gongxiangdanche.__browseClick__
except:
__browseClick__=None
if __browseClick__ =='是':
req_dict['sort']='clicknum'
elif __browseClick__ =='时长':
req_dict['sort']='browseduration'
else:
req_dict['sort']='clicktime'
msg['data']['list'], msg['data']['currPage'], msg['data']['totalPage'], msg['data']['total'],
msg['data']['pageSize'] = gongxiangdanche.page(gongxiangdanche, gongxiangdanche, req_dict)
return jsonify(msg)
# 分页接口（前端）
@main_bp.route("/python501552a0/gongxiangdanche/list", methods=['GET'])
def python501552a0_gongxiangdanche_list():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data":{"currPage":1,"totalPage":1,"total":1,"pageSize":10,"list":[]}}
req_dict = session.get("req_dict")
if req_dict.__contains__('vipread'):
del req_dict['vipread']
userinfo = session.get("params")
try:
__foreEndList__=gongxiangdanche.__foreEndList__
except:
__foreEndList__=None
if __foreEndList__ and __foreEndList__!="否":
tablename=session.get("tablename")
if tablename!="users" and session.get("params")!=None:
req_dict['userid']=session.get("params").get("id")
try:
__foreEndListAuth__=gongxiangdanche.__foreEndListAuth__
except:
__foreEndListAuth__=None
if __foreEndListAuth__ and __foreEndListAuth__!="否":
tablename=session.get("tablename")
if tablename!="users" and session.get("params")!=None:
req_dict['userid']=session.get("params").get("id")
tablename=session.get("tablename")
if tablename=="users" :
try:
del req_dict["userid"]
except:
pass
else:
mapping_str_to_object = {}
for model in Base_model._decl_class_registry.values():
if hasattr(model, '__tablename__'):
mapping_str_to_object[model.__tablename__] = model
try:
__isAdmin__=mapping_str_to_object[tablename].__isAdmin__
except:
__isAdmin__=None
if __isAdmin__!="是" and session.get("params")!=None:
req_dict["userid"]=session.get("params").get("id")
else:
try:
del req_dict["userid"]
except:
pass
if 'luntan' in 'gongxiangdanche':
if 'userid' in req_dict.keys():
del req_dict["userid"]
if 'discuss' in 'gongxiangdanche':
if 'userid' in req_dict.keys():
del req_dict["userid"]
msg['data']['list'], msg['data']['currPage'], msg['data']['totalPage'], msg['data']['total'],
msg['data']['pageSize'] = gongxiangdanche.page(gongxiangdanche, gongxiangdanche, req_dict)
return jsonify(msg)
# 保存接口（后端）
@main_bp.route("/python501552a0/gongxiangdanche/save", methods=['POST'])
def python501552a0_gongxiangdanche_save():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
for key in req_dict:
if req_dict[key] == '':
req_dict[key] = None
error= gongxiangdanche.createbyreq(gongxiangdanche, gongxiangdanche, req_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 添加接口（前端）
@main_bp.route("/python501552a0/gongxiangdanche/add", methods=['POST'])
def python501552a0_gongxiangdanche_add():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
try:
__foreEndListAuth__=gongxiangdanche.__foreEndListAuth__
except:
__foreEndListAuth__=None
if __foreEndListAuth__ and __foreEndListAuth__!="否":
tablename=session.get("tablename")
if tablename!="users":
req_dict['userid']=session.get("params").get("id")
error= gongxiangdanche.createbyreq(gongxiangdanche, gongxiangdanche, req_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 踩、赞接口
@main_bp.route("/python501552a0/gongxiangdanche/thumbsup/<id_>", methods=['GET'])
def python501552a0_gongxiangdanche_thumbsup(id_):
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
id_=int(id_)
type_=int(req_dict.get("type",0))
rets=gongxiangdanche.getbyid(gongxiangdanche, gongxiangdanche,id_)
update_dict={
"id":id_,
}
if type_==1:#赞
update_dict["thumbsupnum"]=int(rets[0].get('thumbsupnum'))+1
elif type_==2:#踩
update_dict["crazilynum"]=int(rets[0].get('crazilynum'))+1
error = gongxiangdanche.updatebyparams(gongxiangdanche, gongxiangdanche, update_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 获取详情信息（后端）
@main_bp.route("/python501552a0/gongxiangdanche/info/<id_>", methods=['GET'])
def python501552a0_gongxiangdanche_info(id_):
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
data = gongxiangdanche.getbyid(gongxiangdanche, gongxiangdanche, int(id_))
if len(data)>0:
msg['data']=data[0]
#浏览点击次数
try:
__browseClick__= gongxiangdanche.__browseClick__
except:
__browseClick__=None
if __browseClick__ and "clicknum" in gongxiangdanche.__table__.columns:
click_dict={"id":int(id_),"clicknum":str(int(data[0].get("clicknum") or 0)+1)}
ret=gongxiangdanche.updatebyparams(gongxiangdanche,gongxiangdanche,click_dict)
if ret!=None:
msg['code'] = crud_error_code
msg['msg'] = ret
return jsonify(msg)
# 获取详情信息（前端）
@main_bp.route("/python501552a0/gongxiangdanche/detail/<id_>", methods=['GET'])
def python501552a0_gongxiangdanche_detail(id_):
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
data = gongxiangdanche.getbyid(gongxiangdanche, gongxiangdanche, int(id_))
if len(data)>0:
msg['data']=data[0]
#浏览点击次数
try:
__browseClick__= gongxiangdanche.__browseClick__
except:
__browseClick__=None
if __browseClick__ and "clicknum" in gongxiangdanche.__table__.columns:
click_dict={"id":int(id_),"clicknum":str(int(data[0].get("clicknum") or 0)+1)}
ret=gongxiangdanche.updatebyparams(gongxiangdanche,gongxiangdanche,click_dict)
if ret!=None:
msg['code'] = crud_error_code
msg['msg'] = ret
return jsonify(msg)
# 更新接口
@main_bp.route("/python501552a0/gongxiangdanche/update", methods=['POST'])
def python501552a0_gongxiangdanche_update():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
if req_dict.get("mima") and "mima" not in gongxiangdanche.__table__.columns :
del req_dict["mima"]
if req_dict.get("password") and "password" not in gongxiangdanche.__table__.columns :
del req_dict["password"]
try:
del req_dict["clicknum"]
except:
pass
error = gongxiangdanche.updatebyparams(gongxiangdanche, gongxiangdanche, req_dict)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 删除接口
@main_bp.route("/python501552a0/gongxiangdanche/delete", methods=['POST'])
def python501552a0_gongxiangdanche_delete():
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
error=gongxiangdanche.delete(
gongxiangdanche,
req_dict
)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
# 投票接口
@main_bp.route("/python501552a0/gongxiangdanche/vote/<int:id_>", methods=['POST'])
def python501552a0_gongxiangdanche_vote(id_):
'''
'''
if request.method == 'POST':
msg = {"code": normal_code, "msg": "success"}
data= gongxiangdanche.getbyid(gongxiangdanche, gongxiangdanche, int(id_))
for i in data:
votenum=i.get('votenum')
if votenum!=None:
params={"id":int(id_),"votenum":votenum+1}
error=gongxiangdanche.updatebyparams(gongxiangdanche,gongxiangdanche,params)
if error!=None:
msg['code'] = crud_error_code
msg['msg'] = error
return jsonify(msg)
@main_bp.route("/python501552a0/gongxiangdanche/sectionStat/dianzanshu", methods=['GET'])
def python501552a0_gongxiangdanche_sectionStat_dianzanshu():
'''
分段统计接口
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": []}
where = " where 1=1"
tablename=session.get("tablename")
sql = """
SELECT '五十以下' as dianzanshu,case when t.五十以下 is null then 0 else t.五十以下 end total
from
(select
sum(case when dianzanshu <= 50 then 1 else 0 end) as 五十以下, sum(case when dianzanshu >= 51 and dianzanshu <= 100 then 1 else 0 end) as 五十到一百, sum(case when dianzanshu >= 101 and dianzanshu <= 150 then 1 else 0 end) as 一百到一百五, sum(case when dianzanshu >= 150 then 1 else 0 end) as 一百五以上 from gongxiangdanche """ +where+""" ) t union all SELECT '五十到一百' as dianzanshu,case when t.五十到一百 is null then 0 else t.五十到一百 end total
from
(select
sum(case when dianzanshu <= 50 then 1 else 0 end) as 五十以下, sum(case when dianzanshu >= 51 and dianzanshu <= 100 then 1 else 0 end) as 五十到一百, sum(case when dianzanshu >= 101 and dianzanshu <= 150 then 1 else 0 end) as 一百到一百五, sum(case when dianzanshu >= 150 then 1 else 0 end) as 一百五以上 from gongxiangdanche """ +where+""" ) t union all SELECT '一百到一百五' as dianzanshu,case when t.一百到一百五 is null then 0 else t.一百到一百五 end total
from
(select
sum(case when dianzanshu <= 50 then 1 else 0 end) as 五十以下, sum(case when dianzanshu >= 51 and dianzanshu <= 100 then 1 else 0 end) as 五十到一百, sum(case when dianzanshu >= 101 and dianzanshu <= 150 then 1 else 0 end) as 一百到一百五, sum(case when dianzanshu >= 150 then 1 else 0 end) as 一百五以上 from gongxiangdanche """ +where+""" ) t union all SELECT '一百五以上' as dianzanshu,case when t.一百五以上 is null then 0 else t.一百五以上 end total
from
(select
sum(case when dianzanshu <= 50 then 1 else 0 end) as 五十以下, sum(case when dianzanshu >= 51 and dianzanshu <= 100 then 1 else 0 end) as 五十到一百, sum(case when dianzanshu >= 101 and dianzanshu <= 150 then 1 else 0 end) as 一百到一百五, sum(case when dianzanshu >= 150 then 1 else 0 end) as 一百五以上 from gongxiangdanche """ +where+""" ) t """
data = db.session.execute(sql)
data = data.fetchall()
results = []
for i in range(len(data)):
result = {
'dianzanshu': decimalEncoder(data[i][0]),
'total': decimalEncoder(data[i][1])
}
results.append(result)
msg['data'] = results
return jsonify(msg)
# 分组统计接口
@main_bp.route("/python501552a0/gongxiangdanche/group/<columnName>", methods=['GET'])
def python501552a0_gongxiangdanche_group(columnName):
'''
分组统计接口
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
userinfo = session.get("params")
msg['data'] = gongxiangdanche.groupbycolumnname(gongxiangdanche,gongxiangdanche,columnName,req_dict)
msg['data'] = msg['data'][:10]
json_filename='gongxiangdanche'+f'_group_{columnName}.json'
where = ' where 1 = 1 '
sql = "SELECT COUNT(*) AS total, " + columnName + " FROM gongxiangdanche " + where + " GROUP BY " + columnName
app.executor.submit(spark_read_mysql, f"({sql}) "+'gongxiangdanche', json_filename)
with open(json_filename, 'w', encoding='utf-8') as f:
f.write(json.dumps(msg['data'], indent=4, ensure_ascii=False))
app.executor.submit(upload_to_hdfs, json_filename)
app.executor.submit(MRMySQLAvg.run)
return jsonify(msg)
# 按值统计接口
@main_bp.route("/python501552a0/gongxiangdanche/value/<xColumnName>/<yColumnName>", methods=['GET'])
def python501552a0_gongxiangdanche_value(xColumnName, yColumnName):
'''
按值统计接口,
{
"code": 0,
"data": [
{
"total": 10.0,
"shangpinleibie": "aa"
},
{
"total": 20.0,
"shangpinleibie": "bb"
},
{
"total": 15.0,
"shangpinleibie": "cc"
}
]
}
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
req_dict = session.get("req_dict")
userinfo = session.get("params")
msg['data'] = gongxiangdanche.getvaluebyxycolumnname(gongxiangdanche,gongxiangdanche,xColumnName,yColumnName,req_dict)
msg['data'] = msg['data'][:10]
return jsonify(msg)
# 按日期统计接口
@main_bp.route("/python501552a0/gongxiangdanche/value/<xColumnName>/<yColumnName>/<timeStatType>", methods=['GET'])
def python501552a0_gongxiangdanche_value_riqi(xColumnName, yColumnName, timeStatType):
'''
按日期统计接口
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": {}}
userinfo = session.get("params")
where = ' where 1 = 1 '
sql = ''
if timeStatType == '日':
sql = "SELECT DATE_FORMAT({0}, '%Y-%m-%d') {0}, sum({1}) total FROM gongxiangdanche {2} GROUP BY DATE_FORMAT({0}, '%Y-%m-%d')".format(xColumnName, yColumnName, where, '%Y-%m-%d')
if timeStatType == '月':
sql = "SELECT DATE_FORMAT({0}, '%Y-%m') {0}, sum({1}) total FROM gongxiangdanche {2} GROUP BY DATE_FORMAT({0}, '%Y-%m')".format(xColumnName, yColumnName, where, '%Y-%m')
if timeStatType == '年':
sql = "SELECT DATE_FORMAT({0}, '%Y') {0}, sum({1}) total FROM gongxiangdanche {2} GROUP BY DATE_FORMAT({0}, '%Y')".format(xColumnName, yColumnName, where, '%Y')
data = db.session.execute(sql)
data = data.fetchall()
results = []
for i in range(len(data)):
result = {
xColumnName: decimalEncoder(data[i][0]),
'total': decimalEncoder(data[i][1])
}
results.append(result)
msg['data'] = results
json_filename='gongxiangdanche'+f'_value_{xColumnName}_{yColumnName}.json'
app.executor.submit(spark_read_mysql, f"({sql}) "+'gongxiangdanche', json_filename)
with open(json_filename, 'w', encoding='utf-8') as f:
f.write(json.dumps(results, indent=4, ensure_ascii=False))
app.executor.submit(upload_to_hdfs, json_filename)
app.executor.submit(MRMySQLAvg.run)
return jsonify(msg)
# 按值统计（多）
@main_bp.route("/python501552a0/gongxiangdanche/valueMul/<xColumnName>", methods=['GET'])
def python501552a0_gongxiangdanche_valueMul(xColumnName):
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": []}
req_dict = session.get("req_dict")
userinfo = session.get("params")
where = ' where 1 = 1 '
for item in req_dict['yColumnNameMul'].split(','):
sql = "SELECT {0}, sum({1}) AS total FROM gongxiangdanche {2} GROUP BY {0} LIMIT 10".format(xColumnName, item, where)
L = []
data = db.session.execute(sql)
data = data.fetchall()
for i in range(len(data)):
result = {
xColumnName: decimalEncoder(data[i][0]),
'total': decimalEncoder(data[i][1])
}
L.append(result)
msg['data'].append(L)
return jsonify(msg)
# 按值统计（多）
@main_bp.route("/python501552a0/gongxiangdanche/valueMul/<xColumnName>/<timeStatType>", methods=['GET'])
def python501552a0_gongxiangdanche_valueMul_time(xColumnName):
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": []}
req_dict = session.get("req_dict")
userinfo = session.get("params")
timeStatType = req_dict['timeStatType']
where = ' where 1 = 1 '
for item in req_dict['yColumnNameMul'].split(','):
sql = ''
if timeStatType == '日':
sql = "SELECT DATE_FORMAT({0}, '%Y-%m-%d') {0}, sum({1}) total FROM gongxiangdanche {2} GROUP BY DATE_FORMAT({0}, '%Y-%m-%d') LIMIT 10".format(xColumnName, item, where, '%Y-%m-%d')
if timeStatType == '月':
sql = "SELECT DATE_FORMAT({0}, '%Y-%m') {0}, sum({1}) total FROM gongxiangdanche {2} GROUP BY DATE_FORMAT({0}, '%Y-%m') LIMIT 10".format(xColumnName, item, where, '%Y-%m')
if timeStatType == '年':
sql = "SELECT DATE_FORMAT({0}, '%Y') {0}, sum({1}) total FROM gongxiangdanche {2} GROUP BY DATE_FORMAT({0}, '%Y') LIMIT 10".format(xColumnName, item, where, '%Y')
L = []
data = db.session.execute(sql)
data = data.fetchall()
for i in range(len(data)):
result = {
xColumnName: decimalEncoder(data[i][0]),
'total': decimalEncoder(data[i][1])
}
L.append(result)
msg['data'].append(L)
return jsonify(msg)
# 总数量
@main_bp.route("/python501552a0/gongxiangdanche/count", methods=['GET'])
def python501552a0_gongxiangdanche_count():
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": 0}
req_dict = session.get("req_dict")
userinfo = session.get("params")
msg['data'] = gongxiangdanche.count(gongxiangdanche, gongxiangdanche, req_dict)
return jsonify(msg)
# 统计接口
@main_bp.route("/python501552a0/gongxiangdanche/remind/<columnName>/<type>", methods=['GET']) #
def python501552a0_gongxiangdanche_remind(columnName,type):
'''
'''
if request.method == 'GET':
msg = {"code": normal_code, 'count': 0}
# 组合查询参数
params = session.get("req_dict")
remindstart = 0
remindend =9999990
if int(type)==1:#数字
if params.get('remindstart') == None and params.get('remindend') != None:
remindstart = 0
remindend = int(params['remindend'])
elif params.get('remindstart') != None and params.get('remindend') == None:
remindstart = int(params['remindstart'])
remindend = 999999
elif params.get('remindstart') == None and params.get('remindend') == None:
remindstart = 0
remindend = 999999
else:
remindstart = params.get('remindstart')
remindend = params.get('remindend')
elif int(type)==2:#日期
current_time=int(time.time())
if params.get('remindstart') == None and params.get('remindend') != None:
starttime=current_time-60*60*24*365*2
params['remindstart'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(starttime))
endtime=current_time+60*60*24*params.get('remindend')
params['remindend'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(endtime))
elif params.get('remindstart') != None and params.get('remindend') == None:
starttime= current_time - 60 * 60 * 24 * params.get('remindstart')
params['remindstart']=time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(starttime))
endtime=current_time+60*60*24*365*2
params['remindend'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(endtime))
elif params.get('remindstart') == None and params.get('remindend') == None:
starttime = current_time - 60 * 60 * 24 * 365 * 2
params['remindstart'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(starttime))
endtime = current_time + 60 * 60 * 24 * 365 * 2
params['remindend'] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(endtime))
data = gongxiangdanche.getbetweenparams(
gongxiangdanche,
gongxiangdanche,
columnName,
{
"remindStart": remindstart,
"remindEnd": remindend
}
)
msg['count'] = len(data)
return jsonify(msg)
#分类列表
@main_bp.route("/python501552a0/gongxiangdanche/lists", methods=['GET'])
def python501552a0_gongxiangdanche_lists():
if request.method == 'GET':
msg = {"code": normal_code, "msg": "success", "data": []}
list,_,_,_,_ = gongxiangdanche.page(gongxiangdanche,gongxiangdanche,{})
msg['data'] = list
return jsonify(msg)
1-项目均为博主学习开发自研，适合新手入门和学习使用
2-所有源码均一手开发，不是模版！不容易跟班里人重复！
源码获取
文章下方名片联系我即可~
大家点赞、收藏、关注、评论啦 、查看👇🏻获取联系方式👇🏻
精彩专栏推荐订阅：在下方专栏
风语者！平时喜欢研究各种技术，目前在从事后端开发工作，热爱生活、热爱工作。
站长推荐
- U8W/U8W-Mini使用与常见问题解决
U8W/U8W-Mini使用与常见问题解决 - 分享几个国内免费的ChatGPT镜像网址(亲测有效)
分享几个国内免费的ChatGPT镜像网址(亲测有效) - stm32使用HAL库配置串口中断收发数据（保姆级教程）
stm32使用HAL库配置串口中断收发数据（保姆级教程） - QT多线程的5种用法，通过使用线程解决UI主界面的耗时操作代码，防止界面卡死。
QT多线程的5种用法，通过使用线程解决UI主界面的耗时操作代码，防止界面卡死。... - SpringSecurity实现前后端分离认证授权
SpringSecurity实现前后端分离认证授权