抓取数据任务编号code

最新推荐文章于 2025-05-15 14:14:02 发布

原创最新推荐文章于 2025-05-15 14:14:02 发布 · 655 阅读

0 ·

CC 4.0 BY-SA版权

sqlserver 专栏收录该内容

58 篇文章

订阅专栏

CODE MODIFYDATE CREATEDATE CURRENTNO
CMAQ\20160225\station\domain3\500000\500000001_20160225.txt2016-03-04 12:01:25.6832016-03-04 12:01:25.683true

这样每解析一个txt就会生成一个code，数据量特别大，查询起来也很不方便

想改成如下这种，同一种数据解析入库的时候code不变，只改变CURRENTNO

CODE MODIFYDATE CREATEDATE CURRENTNO
Hour-_2_2016 2016-03-25 11:44:04.183 2016-01-07 19:12:31.283 2016-3-24 21:00:00

<1>code: CMAQ小时数据任务CMAQJob CMAQ_dmain3_Hour_500000001_2016(CMAQ\20160225\station\domain3\500000\500000001_20160225.txt)

#region 任务编号20160825：CMAQ_dmain3_Hour_500000001_2016
//txt文件名称是：500000001_20160614.txt（synctxtItem.FileName）
int year = DateTime.Now.Year;
string stationcode = synctxtItem.FileName.Split('_')[0];//站点编号
string code = "CMAQ_Hour_" + stationcode + "_" + year;
var apiSchedule = _aqiScheduleRepository.GetAll().FirstOrDefault(x => x.Code == code);
string path = synctxtItem.Path + "\\" + synctxtItem.FileName;
var data = AnalysisText(synctxtItem.FileInfo, synctxtItem.MonitDate, synctxtItem.StationCode, synctxtItem.Domain.ToString());
var daydatas = GetSaveData(data);
foreach (var forecastlDay in daydatas)
{
try
{
_forecastHourRepository.AddOrUpdate(forecastlDay);//添加或更新数据
}
catch (Exception ex)
{
Logger.LogInformation("出错啦:" + ex.ToString());

}
}
Logger.LogDebug("完成文件：" + path);
//添加或更新任务编号
if (apiSchedule == null)
{
apiSchedule = new AqiSchedule()
{
Code = code,
CurrentNo = synctxtItem.FileName.Split('_')[1].Split('.')[0],//获取txt文件的日期20160614
CreateDate = DateTime.Now,
ModifyDate = DateTime.Now
};

_aqiScheduleRepository.AddOrUpdate(apiSchedule);
}
else
{
apiSchedule.ModifyDate = DateTime.Now;
apiSchedule.CurrentNo = synctxtItem.FileName.Split('_')[1].Split('.')[0];//获取txt文件的日期20160614
_aqiScheduleRepository.Update(apiSchedule);
}

#endregion

<1.1>以上代码的问题是，不管数据库中是否已经有txt里的数据，都会进入到这里：

try
{
_forecastHourRepository.AddOrUpdate(forecastlDay);//添加或更新数据
}
catch (Exception ex)
{
Logger.LogInformation("出错啦:" + ex.ToString());

}

<1.2>当txt文件比较多的时候，肯定会影响代码效率，所以修改如下：

//添加或更新任务编号
if (apiSchedule == null)
{
foreach (var forecastlDay in daydatas)
{
try
{
_forecastHourRepository.AddOrUpdate(forecastlDay);//添加或更新数据
}
catch (Exception ex)
{
Logger.LogInformation("出错啦:" + ex.ToString());

}
}
Logger.LogDebug("完成文件：" + path);
apiSchedule = new AqiSchedule()
{
Code = code,
CurrentNo = txtdate,//获取txt文件的日期20160614
CreateDate = DateTime.Now,
ModifyDate = DateTime.Now
};

_aqiScheduleRepository.AddOrUpdate(apiSchedule);

}
else
{
DateTime txt_date= DateTime.ParseExact(txtdate, "yyyyMMdd", System.Globalization.CultureInfo.CurrentCulture);//txt日期
DateTime no_date = DateTime.ParseExact(apiSchedule.CurrentNo, "yyyyMMdd", System.Globalization.CultureInfo.CurrentCulture);//已经入库的txt日期
//txt的日期大于>CurrentNo,进行数据入库操作
if (txt_date > no_date) {
foreach (var forecastlDay in daydatas)
{
try
{
_forecastHourRepository.AddOrUpdate(forecastlDay);//添加或更新数据
}
catch (Exception ex)
{
Logger.LogInformation("出错啦:" + ex.ToString());

}
}
Logger.LogDebug("完成文件：" + path);
}
apiSchedule.ModifyDate = DateTime.Now;
apiSchedule.CurrentNo = txtdate;//获取txt文件的日期20160614
_aqiScheduleRepository.Update(apiSchedule);
}

<1.3>以上代码的问题是：这里只取到了‘20160616\station\domain3\500000’里的文件

但没取到'20160616\station\domain3\500000'里的文件

因为 txt_date一样

当domain4文件要入库时，if (txt_date > no_date)里的条件表达式为false

<1.4>修改如下：

string domaintype = synctxtItem.Type;//获取domain文件夹名称

string code = "CMAQ_"+ domaintype + "_Hour_" + stationcode + "_" + year;

<2>code: CMAQ日数据任务CMAQDayJob CMAQ_dmain3_Day_500000001_2016(CMAQ\20160630\station\domain3\500000\500000004_20160630.txt-day)

<3>code: wrfchem小时数据任务LvLiangWrfChemTxtHourJob wrfchem_dmain3_Hour_500000001_2016(wrfchem\20160225\station\domain3\500000\500000001_20160225.txt)

<4>code: wrfchem日数据任务LvLiangWrfChemTxtDayJob wrfchem_dmain3_Day_500000001_2016(wrfchem\20160225\station\domain3\500000\500000001_20160225.txt-day)

<5>code: statistic小时数据任务StatisticalForecastTxtHourJob

statistic_Hour_500000001_2016(statistic_output\20160404\hour\500000\500000001_statistic_forecasting_72hours_20160404.txt)