Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..XX.......................................X....X..........] duration: 5.956583379s FAIL (2025-08-08 00:04:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.9566043s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.865314477s FAIL (2025-08-08 00:04:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.865339745s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................................XX..X...........] duration: 1.87148833s FAIL (2025-08-08 00:04:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.871513979s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X............................................XXX.XX....X..] duration: 6.018588553s FAIL (2025-08-08 00:04:18 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.018607479s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................................XX..XXXX....XX......] duration: 7.680171785s FAIL (2025-08-08 00:04:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.700898256s) [upload=14855]: Done starting upload Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments... Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments... Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments... Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment... Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment. Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments... Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces. Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments... Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.680198145s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 648.326673ms target: https://us1.storj.io/login FAIL (2025-08-08 00:04:08 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 648.352974ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XX.X.X.........................................XXXXXX.......] duration: 5.665615643s FAIL (2025-08-08 00:04:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.665631874s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [XXX.X.X.........................................XXXXXX......] duration: 25.519122348s FAIL (2025-08-08 00:03:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.51913356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [X............................................XX..X..........] duration: 1.938122067s FAIL (2025-08-08 00:03:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.938147977s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..XX.......................................X....X.........] duration: 5.958879582s FAIL (2025-08-08 00:03:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.958901353s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..............X........X.....................] duration: 7.526259188s FAIL (2025-08-08 00:03:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 7.526293364s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................................XX..XXXX....XX.....] duration: 6.887168535s FAIL (2025-08-08 00:03:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.700898256s) [upload=14855]: Done starting upload Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments... Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments... Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments... Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment... Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment. Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments... Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces. Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments... Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.88719736s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [..........................................XXXX...XXXXX......] duration: 1.849512424s FAIL (2025-08-08 00:03:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:40.612588843 (14h15m40.20193383s) [upload=14781]: Starting upload Aug 7 23:55:40.612648929 (14h15m40.201993906s) [upload=14781]: Done starting upload Aug 7 23:55:40.612717029 (14h15m40.202062016s) [upload=14781]: Uploading segments... Aug 7 23:55:40.612750823 (14h15m40.20209581s) [upload=14781]: Got next segment. Inline: true Aug 7 23:55:40.612754510 (14h15m40.202099487s) [upload=14781]: Waiting for error group managing segments... Aug 7 23:55:40.612786883 (14h15m40.20213186s) [upload=14781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:40.612790891 (14h15m40.202135868s) [upload=14781]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:40.612793075 (14h15m40.202138062s) [upload=14781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:40.883469345 (14h15m40.472814333s) [upload=14781]: Done uploading segments... Aug 7 23:55:40.883486387 (14h15m40.472831375s) [upload=14781]: Upload finished. err: <nil> Aug 7 23:56:17.423712239 (14h16m17.013057226s) [upload=14789]: Starting upload Aug 7 23:56:17.423767895 (14h16m17.013112882s) [upload=14789]: Done starting upload Aug 7 23:56:17.423814835 (14h16m17.013159822s) [upload=14789]: Uploading segments... Aug 7 23:56:17.423847678 (14h16m17.013192665s) [upload=14789]: Got next segment. Inline: true Aug 7 23:56:17.423851335 (14h16m17.013196322s) [upload=14789]: Waiting for error group managing segments... Aug 7 23:56:17.423883377 (14h16m17.013228364s) [upload=14789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:17.423887054 (14h16m17.013232041s) [upload=14789]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:17.423889258 (14h16m17.013234245s) [upload=14789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:17.689327115 (14h16m17.278672102s) [upload=14789]: Done uploading segments... Aug 7 23:56:17.689345711 (14h16m17.278690688s) [upload=14789]: Upload finished. err: <nil> Aug 7 23:56:58.288741281 (14h16m57.878086268s) [upload=14801]: Starting upload Aug 7 23:56:58.288797098 (14h16m57.878142085s) [upload=14801]: Done starting upload Aug 7 23:56:58.288851010 (14h16m57.878195987s) [upload=14801]: Uploading segments... Aug 7 23:56:58.288882281 (14h16m57.878227268s) [upload=14801]: Got next segment. Inline: true Aug 7 23:56:58.288887911 (14h16m57.878232898s) [upload=14801]: Waiting for error group managing segments... Aug 7 23:56:58.288919792 (14h16m57.878264769s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.288923970 (14h16m57.878268957s) [upload=14801]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.288926485 (14h16m57.878271462s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:58.556529816 (14h16m58.145874803s) [upload=14801]: Done uploading segments... Aug 7 23:56:58.556544975 (14h16m58.145889962s) [upload=14801]: Upload finished. err: <nil> Aug 7 23:57:18.362794018 (14h17m17.952138995s) [upload=14806]: Starting upload Aug 7 23:57:18.362883679 (14h17m17.952228666s) [upload=14806]: Done starting upload Aug 7 23:57:18.362928104 (14h17m17.952273092s) [upload=14806]: Uploading segments... Aug 7 23:57:18.362957320 (14h17m17.952302307s) [upload=14806]: Got next segment. Inline: true Aug 7 23:57:18.362961077 (14h17m17.952306064s) [upload=14806]: Waiting for error group managing segments... Aug 7 23:57:18.363018407 (14h17m17.952363394s) [upload=14806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:18.363022404 (14h17m17.952367392s) [upload=14806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:18.363024739 (14h17m17.952369716s) [upload=14806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:47.715297444 (14h17m47.304642431s) [upload=14806]: Done uploading segments... Aug 7 23:57:47.715410370 (14h17m47.304755357s) [upload=14806]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:53.375540080 (14h17m52.964885068s) [upload=14813]: Starting upload Aug 7 23:57:53.375600245 (14h17m52.964945232s) [upload=14813]: Done starting upload Aug 7 23:57:53.375654389 (14h17m52.964999376s) [upload=14813]: Uploading segments... Aug 7 23:57:53.375686622 (14h17m52.965031609s) [upload=14813]: Got next segment. Inline: true Aug 7 23:57:53.375690759 (14h17m52.965035746s) [upload=14813]: Waiting for error group managing segments... Aug 7 23:57:53.375725876 (14h17m52.965070863s) [upload=14813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:53.375730134 (14h17m52.965075121s) [upload=14813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:53.375734372 (14h17m52.965079359s) [upload=14813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:22.726838464 (14h18m22.316183451s) [upload=14813]: Done uploading segments... Aug 7 23:58:22.726860075 (14h18m22.316205062s) [upload=14813]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.621870384 (14h18m23.211215371s) [upload=14822]: Starting upload Aug 7 23:58:23.621952000 (14h18m23.211296987s) [upload=14822]: Done starting upload Aug 7 23:58:23.622003359 (14h18m23.211348346s) [upload=14822]: Uploading segments... Aug 7 23:58:23.622043436 (14h18m23.211388413s) [upload=14822]: Got next segment. Inline: true Aug 7 23:58:23.622046772 (14h18m23.211391759s) [upload=14822]: Waiting for error group managing segments... Aug 7 23:58:23.622078573 (14h18m23.21142356s) [upload=14822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.622082230 (14h18m23.211427217s) [upload=14822]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.622084534 (14h18m23.211429511s) [upload=14822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:52.983677879 (14h18m52.573022876s) [upload=14822]: Done uploading segments... Aug 7 23:58:52.983713708 (14h18m52.573058685s) [upload=14822]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:11.446858470 (14h19m11.036203457s) [upload=14835]: Starting upload Aug 7 23:59:11.446906493 (14h19m11.03625148s) [upload=14835]: Done starting upload Aug 7 23:59:11.446947491 (14h19m11.036292479s) [upload=14835]: Uploading segments... Aug 7 23:59:11.446975475 (14h19m11.036320462s) [upload=14835]: Got next segment. Inline: true Aug 7 23:59:11.446978751 (14h19m11.036323738s) [upload=14835]: Waiting for error group managing segments... Aug 7 23:59:11.447008889 (14h19m11.036353876s) [upload=14835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:11.447013638 (14h19m11.036358625s) [upload=14835]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:11.447016212 (14h19m11.0363612s) [upload=14835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:40.799397635 (14h19m40.388742622s) [upload=14835]: Done uploading segments... Aug 7 23:59:40.799413266 (14h19m40.388758243s) [upload=14835]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:59.523080643 (14h19m59.11242563s) [upload=14847]: Starting upload Aug 7 23:59:59.523152089 (14h19m59.112497076s) [upload=14847]: Done starting upload Aug 7 23:59:59.523213727 (14h19m59.112558714s) [upload=14847]: Uploading segments... Aug 7 23:59:59.523251349 (14h19m59.112596336s) [upload=14847]: Got next segment. Inline: true Aug 7 23:59:59.523255287 (14h19m59.112600274s) [upload=14847]: Waiting for error group managing segments... Aug 7 23:59:59.523300013 (14h19m59.112645s) [upload=14847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:59.523304251 (14h19m59.112649238s) [upload=14847]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:59.523307146 (14h19m59.112652133s) [upload=14847]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:28.877190143 (14h20m28.46653516s) [upload=14847]: Done uploading segments... Aug 8 00:00:28.877218477 (14h20m28.466563505s) [upload=14847]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.849542181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [XX..............................................X.XXXX....X.] duration: 1.864516872s FAIL (2025-08-08 00:03:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.864530209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....X.X...................................X.........X.X..X.] duration: 2.289162119s FAIL (2025-08-08 00:03:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.289181756s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 9 out of 60 is failed (threshold=3) [XXX..............................................X.XXXX....X] duration: 1m0.000786671s FAIL (2025-08-08 00:03:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:29.562470053 (14h19m29.15181504s) [upload=14840]: Starting upload Aug 7 23:59:29.562538985 (14h19m29.151883972s) [upload=14840]: Done starting upload Aug 7 23:59:29.562614629 (14h19m29.151959616s) [upload=14840]: Uploading segments... Aug 7 23:59:29.562650589 (14h19m29.151995566s) [upload=14840]: Got next segment. Inline: true Aug 7 23:59:29.562655498 (14h19m29.152000485s) [upload=14840]: Waiting for error group managing segments... Aug 7 23:59:29.562686507 (14h19m29.152031494s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:29.562690264 (14h19m29.152035241s) [upload=14840]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:29.562692428 (14h19m29.152037405s) [upload=14840]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:29.822497550 (14h19m29.411842537s) [upload=14840]: Done uploading segments... Aug 7 23:59:29.822514693 (14h19m29.41185968s) [upload=14840]: Upload finished. err: <nil> Aug 8 00:00:35.820554363 (14h20m35.40989935s) [upload=14856]: Starting upload Aug 8 00:00:35.820610611 (14h20m35.409955598s) [upload=14856]: Done starting upload Aug 8 00:00:35.820669503 (14h20m35.4100145s) [upload=14856]: Uploading segments... Aug 8 00:00:35.820742333 (14h20m35.41008732s) [upload=14856]: Got next segment. Inline: true Aug 8 00:00:35.820747282 (14h20m35.410092269s) [upload=14856]: Waiting for error group managing segments... Aug 8 00:00:35.820788371 (14h20m35.410133358s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.820793050 (14h20m35.410138037s) [upload=14856]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.820795485 (14h20m35.410140472s) [upload=14856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:36.084321816 (14h20m35.673666803s) [upload=14856]: Done uploading segments... Aug 8 00:00:36.084338618 (14h20m35.673683615s) [upload=14856]: Upload finished. err: <nil> Aug 8 00:01:14.496466774 (14h21m14.085811781s) [upload=14868]: Starting upload Aug 8 00:01:14.496556174 (14h21m14.085901162s) [upload=14868]: Done starting upload Aug 8 00:01:14.496609206 (14h21m14.085954183s) [upload=14868]: Uploading segments... Aug 8 00:01:14.496639985 (14h21m14.085984972s) [upload=14868]: Got next segment. Inline: true Aug 8 00:01:14.496643592 (14h21m14.085988579s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:01:14.496678638 (14h21m14.086023625s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:14.496682386 (14h21m14.086027373s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:14.496684681 (14h21m14.086029658s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:14.760681029 (14h21m14.350026016s) [upload=14868]: Done uploading segments... Aug 8 00:01:14.760702580 (14h21m14.350047567s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:01:50.577049881 (14h21m50.166394858s) [upload=14878]: Starting upload Aug 8 00:01:50.577100538 (14h21m50.166445525s) [upload=14878]: Done starting upload Aug 8 00:01:50.577180091 (14h21m50.166525078s) [upload=14878]: Uploading segments... Aug 8 00:01:50.577228012 (14h21m50.166573s) [upload=14878]: Got next segment. Inline: true Aug 8 00:01:50.577231469 (14h21m50.166576456s) [upload=14878]: Waiting for error group managing segments... Aug 8 00:01:50.577270633 (14h21m50.166615621s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:50.577274721 (14h21m50.166619708s) [upload=14878]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:50.577276906 (14h21m50.166621893s) [upload=14878]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:50.863237339 (14h21m50.452582326s) [upload=14878]: Done uploading segments... Aug 8 00:01:50.863254973 (14h21m50.452599961s) [upload=14878]: Upload finished. err: <nil> Aug 8 00:02:41.474648957 (14h22m41.063993944s) [upload=14899]: Starting upload Aug 8 00:02:41.474716297 (14h22m41.064061274s) [upload=14899]: Done starting upload Aug 8 00:02:41.474771743 (14h22m41.06411673s) [upload=14899]: Uploading segments... Aug 8 00:02:41.474801099 (14h22m41.064146086s) [upload=14899]: Got next segment. Inline: true Aug 8 00:02:41.474804966 (14h22m41.064149953s) [upload=14899]: Waiting for error group managing segments... Aug 8 00:02:41.474841336 (14h22m41.064186323s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:41.474845214 (14h22m41.064190201s) [upload=14899]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:41.474847558 (14h22m41.064192535s) [upload=14899]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:03:40.835731182 (14h23m40.425076189s) [upload=14899]: Done uploading segments... Aug 8 00:03:40.835778212 (14h23m40.425123199s) [upload=14899]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000813241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......XX.X.........................................X.XX...X.] duration: 9.411298494s FAIL (2025-08-08 00:03:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces. Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments... Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments... Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments... Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded Duration: 9.411315126s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 648.411516ms target: https://us1.storj.io/login FAIL (2025-08-08 00:03:36 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 648.453076ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.X..XX.......................................X....X........] duration: 6.140789039s FAIL (2025-08-08 00:03:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.140808816s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.846466156s FAIL (2025-08-08 00:03:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.846484581s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [X.....X.X...................................X.........X.X..X] duration: 30.001192116s FAIL (2025-08-08 00:03:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001212645s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......XX.X.........................................X.XX...X] duration: 1m0.000770518s FAIL (2025-08-08 00:03:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246137011 (14h21m15.835481998s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.246560380 (14h21m15.835905367s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247408974 (14h21m15.836753961s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247678410 (14h21m15.837023397s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247919882 (14h21m15.837264869s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.247943888 (14h21m15.837288875s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248293266 (14h21m15.837638253s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248420540 (14h21m15.837765527s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248557311 (14h21m15.837902288s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248758506 (14h21m15.838103493s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248897792 (14h21m15.838242779s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.248989629 (14h21m15.838334616s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249186575 (14h21m15.838531552s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249273772 (14h21m15.838618759s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249459598 (14h21m15.838804585s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:16.249717240 (14h21m15.839062227s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Uploading piece... Aug 8 00:01:17.413257645 (14h21m17.002602632s) [upload=14870]: Next returned nil segment Aug 8 00:01:17.413273324 (14h21m17.002618311s) [upload=14870]: Waiting for error group managing segments... Aug 8 00:01:17.710897294 (14h21m17.300242281s) [noise=true , address=64.49.13.169:4705 , piece_id=3MVXPRNKBBUSYAH2E7PCIWXDCUFQQ547IPOYIDLQ3T7QDYPBJA7A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:17.715058896 (14h21m17.304403883s) [noise=true , address=64.49.13.139:4471 , piece_id=FPVXYIRNZXODU7N5RPCDOVVUECXWRTXCLINEXMOJTKNHF6YN46YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.126218161 (14h21m17.715563148s) [noise=true , address=79.127.201.215:4356 , piece_id=VWNEIHNI4KPPF4QLW36INIWTUI2CD6MZYSDFBHFTDOKPS73M57QA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.139272367 (14h21m17.728617354s) [noise=true , address=79.127.219.51:4181 , piece_id=4LYG6AL2BWD3UYQQAUUGZNJ7HUFXSHZ2TULWJOYSWTJLNVGHVKLQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.221269031 (14h21m17.810614018s) [noise=true , address=104.36.82.43:4855 , piece_id=MJYC5ZGE6P72UYNYDRMH3BAZ6KIY57SA3EKJR54W6HPZNO3AE4IA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.257350144 (14h21m17.846695121s) [noise=true , address=104.36.82.37:4659 , piece_id=627LA2AVL53NXFMDWSLGBO5GUKDAIGGSOPBFRPYIPYC5EAQC6P7Q, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.258435089 (14h21m17.847780076s) [noise=true , address=104.36.82.44:4876 , piece_id=BA2G42OXQD3NLONUVD475S2BNC7T37T5PHPE3K3E5HJAJP4W5MKQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.287010923 (14h21m17.87635591s) [noise=true , address=104.36.82.7:4920 , piece_id=BN5NCMLEGDVRIHO6YOWM5FNPCGFYC3BDKLOU7DIP6K6DYZXJ5A5A, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.348793001 (14h21m17.938137988s) [noise=true , address=104.36.82.11:4143 , piece_id=E4AW52SZYZVRWPU4L56INK3IRNG7UNO5RRIRJANLMEPZQVDUHCEQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.378129459 (14h21m17.967474436s) [noise=true , address=104.36.82.58:4275 , piece_id=VBV72SK36XIHMWSZTJJE4MZPLD5VZ6Z6UPBHK4A2ZI2C3O3RCVXA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.386414901 (14h21m17.975759888s) [noise=true , address=104.36.82.46:4487 , piece_id=6BZYNRXKBHFL3RQTGN3ZN4RV2O4USZ24GGCVSKFZWNS6N32L3NGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.516150823 (14h21m18.10549581s) [noise=true , address=104.36.82.9:4024 , piece_id=FM7MVIPG7C264O6Q64RUFOKZ2E5NMTIPWSIVMH5SWZUSCMFI7UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:18.626647204 (14h21m18.215992191s) [noise=true , address=77.90.31.14:4672 , piece_id=3ZYE5SLRNLY2ALXRR4QGCZT4SQ37U7FIAUOCVT63HQSBKPKV4UGQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042108811 (14h21m18.631453798s) [noise=true , address=104.36.82.22:4512 , piece_id=7VOWVUPU3QVXIKML65NHX3NL7AC26QSDGHSSCRLUXIVZRNUPPKRA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: <nil> Aug 8 00:01:19.042136654 (14h21m18.631481641s) [seg_pos={0 0}, upload=14870]: Segment reached optimal threshold of 14 pieces. Aug 8 00:01:19.042375541 (14h21m18.631720518s) [noise=true , address=45.91.116.5:4302 , piece_id=6RMPHN4LYSZVM6XKAUGSU7GY3TIKBGLKK4KNFMZQ2FMVEEVWV2LA, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:19.042485912 (14h21m18.6318309s) [noise=true , address=104.36.82.55:4209 , piece_id=PJ4RK52LARBDG6WEVKXPZPX3RKJVIRSDCZY3B6CVCIUZO6BTWLBQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR): protocol: rpc: dial tcp 104.36.82.55:4209: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042532361 (14h21m18.631877348s) [noise=true , address=104.36.82.20:4466 , piece_id=PLG7Z2WTYKUNB347NPT4ESIDK2NHCP66NPZJOXYL4A6L5OMFQIOQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqNLPRWY7fJRS6F9h4B4qVFv6U5jyWkCy3Um7jfjEdCShEjsD): protocol: write tcp 188.166.225.170:37056->104.36.82.20:4466: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:01:19.042616462 (14h21m18.631961449s) [noise=true , address=104.36.82.5:4844 , piece_id=REDGG4W6TJYEPEEPR65O5AVQDU2QMBNRA6HQQRUJA6JP7IS5T7YQ, seg_pos={0 0}, upload=14870]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iovpNdbidkiv9mEHaAPjBubGmoqr4sUJWm8Cb64LzS6wuwLWa): protocol: rpc: dial tcp 104.36.82.5:4844: operation was canceled; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:01:19.042628876 (14h21m18.631973863s) [seg_pos={0 0}, upload=14870]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:01:19.042699029 (14h21m18.632044017s) [upload=14870]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:19.042703128 (14h21m18.632048115s) [upload=14870]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:19.344335316 (14h21m18.933680293s) [upload=14870]: Done uploading segments... Aug 8 00:01:19.344353089 (14h21m18.933698077s) [upload=14870]: Upload finished. err: <nil> Aug 8 00:02:30.778225454 (14h22m30.367570441s) [upload=14892]: Starting upload Aug 8 00:02:30.778283114 (14h22m30.367628151s) [upload=14892]: Done starting upload Aug 8 00:02:30.778411841 (14h22m30.367756828s) [upload=14892]: Uploading segments... Aug 8 00:02:30.778539616 (14h22m30.367884623s) [upload=14892]: Got next segment. Inline: false Aug 8 00:02:30.778583389 (14h22m30.367928376s) [upload=14892]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:30.778587717 (14h22m30.367932704s) [upload=14892]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:30.135980490 (14h23m29.725325467s) [upload=14892]: Next segment err: context deadline exceeded Aug 8 00:03:30.136177137 (14h23m29.725522124s) [upload=14892]: Done uploading segments... Aug 8 00:03:30.136196834 (14h23m29.725541821s) [upload=14892]: Upload finished. err: context deadline exceeded Duration: 1m0.000793702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X..............X........X....................] duration: 8.082911061s FAIL (2025-08-08 00:03:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 8.082941309s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X............................................XXX.XX....X.] duration: 5.923997484s FAIL (2025-08-08 00:03:26 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.924018133s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................................XX..XXXX....XX....] duration: 6.693453032s FAIL (2025-08-08 00:03:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.700898256s) [upload=14855]: Done starting upload Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments... Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments... Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments... Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment... Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment. Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments... Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces. Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments... Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.693484101s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X............................................XXX.XX....X] duration: 30.000798398s FAIL (2025-08-08 00:03:20 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000818798s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [...........................................XXXX...XXXXX.....] duration: 1.865844883s FAIL (2025-08-08 00:03:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:40.612588843 (14h15m40.20193383s) [upload=14781]: Starting upload Aug 7 23:55:40.612648929 (14h15m40.201993906s) [upload=14781]: Done starting upload Aug 7 23:55:40.612717029 (14h15m40.202062016s) [upload=14781]: Uploading segments... Aug 7 23:55:40.612750823 (14h15m40.20209581s) [upload=14781]: Got next segment. Inline: true Aug 7 23:55:40.612754510 (14h15m40.202099487s) [upload=14781]: Waiting for error group managing segments... Aug 7 23:55:40.612786883 (14h15m40.20213186s) [upload=14781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:40.612790891 (14h15m40.202135868s) [upload=14781]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:40.612793075 (14h15m40.202138062s) [upload=14781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:40.883469345 (14h15m40.472814333s) [upload=14781]: Done uploading segments... Aug 7 23:55:40.883486387 (14h15m40.472831375s) [upload=14781]: Upload finished. err: <nil> Aug 7 23:56:17.423712239 (14h16m17.013057226s) [upload=14789]: Starting upload Aug 7 23:56:17.423767895 (14h16m17.013112882s) [upload=14789]: Done starting upload Aug 7 23:56:17.423814835 (14h16m17.013159822s) [upload=14789]: Uploading segments... Aug 7 23:56:17.423847678 (14h16m17.013192665s) [upload=14789]: Got next segment. Inline: true Aug 7 23:56:17.423851335 (14h16m17.013196322s) [upload=14789]: Waiting for error group managing segments... Aug 7 23:56:17.423883377 (14h16m17.013228364s) [upload=14789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:17.423887054 (14h16m17.013232041s) [upload=14789]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:17.423889258 (14h16m17.013234245s) [upload=14789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:17.689327115 (14h16m17.278672102s) [upload=14789]: Done uploading segments... Aug 7 23:56:17.689345711 (14h16m17.278690688s) [upload=14789]: Upload finished. err: <nil> Aug 7 23:56:58.288741281 (14h16m57.878086268s) [upload=14801]: Starting upload Aug 7 23:56:58.288797098 (14h16m57.878142085s) [upload=14801]: Done starting upload Aug 7 23:56:58.288851010 (14h16m57.878195987s) [upload=14801]: Uploading segments... Aug 7 23:56:58.288882281 (14h16m57.878227268s) [upload=14801]: Got next segment. Inline: true Aug 7 23:56:58.288887911 (14h16m57.878232898s) [upload=14801]: Waiting for error group managing segments... Aug 7 23:56:58.288919792 (14h16m57.878264769s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.288923970 (14h16m57.878268957s) [upload=14801]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.288926485 (14h16m57.878271462s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:58.556529816 (14h16m58.145874803s) [upload=14801]: Done uploading segments... Aug 7 23:56:58.556544975 (14h16m58.145889962s) [upload=14801]: Upload finished. err: <nil> Aug 7 23:57:18.362794018 (14h17m17.952138995s) [upload=14806]: Starting upload Aug 7 23:57:18.362883679 (14h17m17.952228666s) [upload=14806]: Done starting upload Aug 7 23:57:18.362928104 (14h17m17.952273092s) [upload=14806]: Uploading segments... Aug 7 23:57:18.362957320 (14h17m17.952302307s) [upload=14806]: Got next segment. Inline: true Aug 7 23:57:18.362961077 (14h17m17.952306064s) [upload=14806]: Waiting for error group managing segments... Aug 7 23:57:18.363018407 (14h17m17.952363394s) [upload=14806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:18.363022404 (14h17m17.952367392s) [upload=14806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:18.363024739 (14h17m17.952369716s) [upload=14806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:47.715297444 (14h17m47.304642431s) [upload=14806]: Done uploading segments... Aug 7 23:57:47.715410370 (14h17m47.304755357s) [upload=14806]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:53.375540080 (14h17m52.964885068s) [upload=14813]: Starting upload Aug 7 23:57:53.375600245 (14h17m52.964945232s) [upload=14813]: Done starting upload Aug 7 23:57:53.375654389 (14h17m52.964999376s) [upload=14813]: Uploading segments... Aug 7 23:57:53.375686622 (14h17m52.965031609s) [upload=14813]: Got next segment. Inline: true Aug 7 23:57:53.375690759 (14h17m52.965035746s) [upload=14813]: Waiting for error group managing segments... Aug 7 23:57:53.375725876 (14h17m52.965070863s) [upload=14813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:53.375730134 (14h17m52.965075121s) [upload=14813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:53.375734372 (14h17m52.965079359s) [upload=14813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:22.726838464 (14h18m22.316183451s) [upload=14813]: Done uploading segments... Aug 7 23:58:22.726860075 (14h18m22.316205062s) [upload=14813]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.621870384 (14h18m23.211215371s) [upload=14822]: Starting upload Aug 7 23:58:23.621952000 (14h18m23.211296987s) [upload=14822]: Done starting upload Aug 7 23:58:23.622003359 (14h18m23.211348346s) [upload=14822]: Uploading segments... Aug 7 23:58:23.622043436 (14h18m23.211388413s) [upload=14822]: Got next segment. Inline: true Aug 7 23:58:23.622046772 (14h18m23.211391759s) [upload=14822]: Waiting for error group managing segments... Aug 7 23:58:23.622078573 (14h18m23.21142356s) [upload=14822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.622082230 (14h18m23.211427217s) [upload=14822]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.622084534 (14h18m23.211429511s) [upload=14822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:52.983677879 (14h18m52.573022876s) [upload=14822]: Done uploading segments... Aug 7 23:58:52.983713708 (14h18m52.573058685s) [upload=14822]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:11.446858470 (14h19m11.036203457s) [upload=14835]: Starting upload Aug 7 23:59:11.446906493 (14h19m11.03625148s) [upload=14835]: Done starting upload Aug 7 23:59:11.446947491 (14h19m11.036292479s) [upload=14835]: Uploading segments... Aug 7 23:59:11.446975475 (14h19m11.036320462s) [upload=14835]: Got next segment. Inline: true Aug 7 23:59:11.446978751 (14h19m11.036323738s) [upload=14835]: Waiting for error group managing segments... Aug 7 23:59:11.447008889 (14h19m11.036353876s) [upload=14835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:11.447013638 (14h19m11.036358625s) [upload=14835]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:11.447016212 (14h19m11.0363612s) [upload=14835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:40.799397635 (14h19m40.388742622s) [upload=14835]: Done uploading segments... Aug 7 23:59:40.799413266 (14h19m40.388758243s) [upload=14835]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:59.523080643 (14h19m59.11242563s) [upload=14847]: Starting upload Aug 7 23:59:59.523152089 (14h19m59.112497076s) [upload=14847]: Done starting upload Aug 7 23:59:59.523213727 (14h19m59.112558714s) [upload=14847]: Uploading segments... Aug 7 23:59:59.523251349 (14h19m59.112596336s) [upload=14847]: Got next segment. Inline: true Aug 7 23:59:59.523255287 (14h19m59.112600274s) [upload=14847]: Waiting for error group managing segments... Aug 7 23:59:59.523300013 (14h19m59.112645s) [upload=14847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:59.523304251 (14h19m59.112649238s) [upload=14847]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:59.523307146 (14h19m59.112652133s) [upload=14847]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:28.877190143 (14h20m28.46653516s) [upload=14847]: Done uploading segments... Aug 8 00:00:28.877218477 (14h20m28.466563505s) [upload=14847]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.865872355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.X............................................XX..X.........] duration: 1.885560184s FAIL (2025-08-08 00:02:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.885582967s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX.X..XX.......................................X....X.......] duration: 6.621894066s FAIL (2025-08-08 00:02:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.62191677s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................................XX..XXXX....XX...] duration: 6.765815241s FAIL (2025-08-08 00:02:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.700898256s) [upload=14855]: Done starting upload Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments... Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments... Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments... Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment... Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment. Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments... Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces. Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments... Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.76584032s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [............................................XXXX...XXXXX....] duration: 1.877685997s FAIL (2025-08-08 00:02:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:40.612588843 (14h15m40.20193383s) [upload=14781]: Starting upload Aug 7 23:55:40.612648929 (14h15m40.201993906s) [upload=14781]: Done starting upload Aug 7 23:55:40.612717029 (14h15m40.202062016s) [upload=14781]: Uploading segments... Aug 7 23:55:40.612750823 (14h15m40.20209581s) [upload=14781]: Got next segment. Inline: true Aug 7 23:55:40.612754510 (14h15m40.202099487s) [upload=14781]: Waiting for error group managing segments... Aug 7 23:55:40.612786883 (14h15m40.20213186s) [upload=14781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:40.612790891 (14h15m40.202135868s) [upload=14781]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:40.612793075 (14h15m40.202138062s) [upload=14781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:40.883469345 (14h15m40.472814333s) [upload=14781]: Done uploading segments... Aug 7 23:55:40.883486387 (14h15m40.472831375s) [upload=14781]: Upload finished. err: <nil> Aug 7 23:56:17.423712239 (14h16m17.013057226s) [upload=14789]: Starting upload Aug 7 23:56:17.423767895 (14h16m17.013112882s) [upload=14789]: Done starting upload Aug 7 23:56:17.423814835 (14h16m17.013159822s) [upload=14789]: Uploading segments... Aug 7 23:56:17.423847678 (14h16m17.013192665s) [upload=14789]: Got next segment. Inline: true Aug 7 23:56:17.423851335 (14h16m17.013196322s) [upload=14789]: Waiting for error group managing segments... Aug 7 23:56:17.423883377 (14h16m17.013228364s) [upload=14789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:17.423887054 (14h16m17.013232041s) [upload=14789]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:17.423889258 (14h16m17.013234245s) [upload=14789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:17.689327115 (14h16m17.278672102s) [upload=14789]: Done uploading segments... Aug 7 23:56:17.689345711 (14h16m17.278690688s) [upload=14789]: Upload finished. err: <nil> Aug 7 23:56:58.288741281 (14h16m57.878086268s) [upload=14801]: Starting upload Aug 7 23:56:58.288797098 (14h16m57.878142085s) [upload=14801]: Done starting upload Aug 7 23:56:58.288851010 (14h16m57.878195987s) [upload=14801]: Uploading segments... Aug 7 23:56:58.288882281 (14h16m57.878227268s) [upload=14801]: Got next segment. Inline: true Aug 7 23:56:58.288887911 (14h16m57.878232898s) [upload=14801]: Waiting for error group managing segments... Aug 7 23:56:58.288919792 (14h16m57.878264769s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.288923970 (14h16m57.878268957s) [upload=14801]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.288926485 (14h16m57.878271462s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:58.556529816 (14h16m58.145874803s) [upload=14801]: Done uploading segments... Aug 7 23:56:58.556544975 (14h16m58.145889962s) [upload=14801]: Upload finished. err: <nil> Aug 7 23:57:18.362794018 (14h17m17.952138995s) [upload=14806]: Starting upload Aug 7 23:57:18.362883679 (14h17m17.952228666s) [upload=14806]: Done starting upload Aug 7 23:57:18.362928104 (14h17m17.952273092s) [upload=14806]: Uploading segments... Aug 7 23:57:18.362957320 (14h17m17.952302307s) [upload=14806]: Got next segment. Inline: true Aug 7 23:57:18.362961077 (14h17m17.952306064s) [upload=14806]: Waiting for error group managing segments... Aug 7 23:57:18.363018407 (14h17m17.952363394s) [upload=14806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:18.363022404 (14h17m17.952367392s) [upload=14806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:18.363024739 (14h17m17.952369716s) [upload=14806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:47.715297444 (14h17m47.304642431s) [upload=14806]: Done uploading segments... Aug 7 23:57:47.715410370 (14h17m47.304755357s) [upload=14806]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:53.375540080 (14h17m52.964885068s) [upload=14813]: Starting upload Aug 7 23:57:53.375600245 (14h17m52.964945232s) [upload=14813]: Done starting upload Aug 7 23:57:53.375654389 (14h17m52.964999376s) [upload=14813]: Uploading segments... Aug 7 23:57:53.375686622 (14h17m52.965031609s) [upload=14813]: Got next segment. Inline: true Aug 7 23:57:53.375690759 (14h17m52.965035746s) [upload=14813]: Waiting for error group managing segments... Aug 7 23:57:53.375725876 (14h17m52.965070863s) [upload=14813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:53.375730134 (14h17m52.965075121s) [upload=14813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:53.375734372 (14h17m52.965079359s) [upload=14813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:22.726838464 (14h18m22.316183451s) [upload=14813]: Done uploading segments... Aug 7 23:58:22.726860075 (14h18m22.316205062s) [upload=14813]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.621870384 (14h18m23.211215371s) [upload=14822]: Starting upload Aug 7 23:58:23.621952000 (14h18m23.211296987s) [upload=14822]: Done starting upload Aug 7 23:58:23.622003359 (14h18m23.211348346s) [upload=14822]: Uploading segments... Aug 7 23:58:23.622043436 (14h18m23.211388413s) [upload=14822]: Got next segment. Inline: true Aug 7 23:58:23.622046772 (14h18m23.211391759s) [upload=14822]: Waiting for error group managing segments... Aug 7 23:58:23.622078573 (14h18m23.21142356s) [upload=14822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.622082230 (14h18m23.211427217s) [upload=14822]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.622084534 (14h18m23.211429511s) [upload=14822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:52.983677879 (14h18m52.573022876s) [upload=14822]: Done uploading segments... Aug 7 23:58:52.983713708 (14h18m52.573058685s) [upload=14822]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:11.446858470 (14h19m11.036203457s) [upload=14835]: Starting upload Aug 7 23:59:11.446906493 (14h19m11.03625148s) [upload=14835]: Done starting upload Aug 7 23:59:11.446947491 (14h19m11.036292479s) [upload=14835]: Uploading segments... Aug 7 23:59:11.446975475 (14h19m11.036320462s) [upload=14835]: Got next segment. Inline: true Aug 7 23:59:11.446978751 (14h19m11.036323738s) [upload=14835]: Waiting for error group managing segments... Aug 7 23:59:11.447008889 (14h19m11.036353876s) [upload=14835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:11.447013638 (14h19m11.036358625s) [upload=14835]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:11.447016212 (14h19m11.0363612s) [upload=14835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:40.799397635 (14h19m40.388742622s) [upload=14835]: Done uploading segments... Aug 7 23:59:40.799413266 (14h19m40.388758243s) [upload=14835]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:59.523080643 (14h19m59.11242563s) [upload=14847]: Starting upload Aug 7 23:59:59.523152089 (14h19m59.112497076s) [upload=14847]: Done starting upload Aug 7 23:59:59.523213727 (14h19m59.112558714s) [upload=14847]: Uploading segments... Aug 7 23:59:59.523251349 (14h19m59.112596336s) [upload=14847]: Got next segment. Inline: true Aug 7 23:59:59.523255287 (14h19m59.112600274s) [upload=14847]: Waiting for error group managing segments... Aug 7 23:59:59.523300013 (14h19m59.112645s) [upload=14847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:59.523304251 (14h19m59.112649238s) [upload=14847]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:59.523307146 (14h19m59.112652133s) [upload=14847]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:28.877190143 (14h20m28.46653516s) [upload=14847]: Done uploading segments... Aug 8 00:00:28.877218477 (14h20m28.466563505s) [upload=14847]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.87770808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................................XX..XXXX....XX..] duration: 7.931709467s FAIL (2025-08-08 00:02:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.700898256s) [upload=14855]: Done starting upload Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments... Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments... Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments... Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment... Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment. Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments... Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces. Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments... Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.931737372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [X.X............................................XX..X........] duration: 1.85241113s FAIL (2025-08-08 00:02:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.852431289s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [XXXX.X.X.........................................XXXXXX.....] duration: 5.664858266s FAIL (2025-08-08 00:02:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.664885117s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 642.857069ms target: https://us1.storj.io/login FAIL (2025-08-08 00:02:28 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 642.886264ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..............X........X...................] duration: 5.772915037s FAIL (2025-08-08 00:02:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 5.772939524s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X................................X..........................] duration: 1.800796737s FAIL (2025-08-08 00:02:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 1.800823147s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XXXXX.X.X.........................................XXXXXX....] duration: 26.349558132s FAIL (2025-08-08 00:02:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.349569744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.............................................XXXX...XXXXX...] duration: 7.397038421s FAIL (2025-08-08 00:02:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m24.001855996s) [upload=14742]: Done starting upload Aug 7 23:53:24.412569290 (14h13m24.001914277s) [upload=14742]: Uploading segments... Aug 7 23:53:24.412638803 (14h13m24.001983831s) [upload=14742]: Got next segment. Inline: true Aug 7 23:53:24.412644354 (14h13m24.001989341s) [upload=14742]: Waiting for error group managing segments... Aug 7 23:53:24.412680903 (14h13m24.002025891s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:24.412685442 (14h13m24.002030429s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:24.412688137 (14h13m24.002033114s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:53.772057432 (14h13m53.361402419s) [upload=14742]: Done uploading segments... Aug 7 23:53:53.772119881 (14h13m53.361464908s) [upload=14742]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:30.496202349 (14h14m30.085547336s) [upload=14761]: Starting upload Aug 7 23:54:30.496270259 (14h14m30.085615246s) [upload=14761]: Done starting upload Aug 7 23:54:30.496341565 (14h14m30.085686552s) [upload=14761]: Uploading segments... Aug 7 23:54:30.496378957 (14h14m30.085723944s) [upload=14761]: Got next segment. Inline: true Aug 7 23:54:30.496383836 (14h14m30.085728823s) [upload=14761]: Waiting for error group managing segments... Aug 7 23:54:30.496441346 (14h14m30.085786333s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:30.496446426 (14h14m30.085791413s) [upload=14761]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:30.496449551 (14h14m30.085794539s) [upload=14761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:59.847423574 (14h14m59.436768561s) [upload=14761]: Done uploading segments... Aug 7 23:54:59.847456066 (14h14m59.436801043s) [upload=14761]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:40.612588843 (14h15m40.20193383s) [upload=14781]: Starting upload Aug 7 23:55:40.612648929 (14h15m40.201993906s) [upload=14781]: Done starting upload Aug 7 23:55:40.612717029 (14h15m40.202062016s) [upload=14781]: Uploading segments... Aug 7 23:55:40.612750823 (14h15m40.20209581s) [upload=14781]: Got next segment. Inline: true Aug 7 23:55:40.612754510 (14h15m40.202099487s) [upload=14781]: Waiting for error group managing segments... Aug 7 23:55:40.612786883 (14h15m40.20213186s) [upload=14781]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:40.612790891 (14h15m40.202135868s) [upload=14781]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:40.612793075 (14h15m40.202138062s) [upload=14781]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:40.883469345 (14h15m40.472814333s) [upload=14781]: Done uploading segments... Aug 7 23:55:40.883486387 (14h15m40.472831375s) [upload=14781]: Upload finished. err: <nil> Aug 7 23:56:17.423712239 (14h16m17.013057226s) [upload=14789]: Starting upload Aug 7 23:56:17.423767895 (14h16m17.013112882s) [upload=14789]: Done starting upload Aug 7 23:56:17.423814835 (14h16m17.013159822s) [upload=14789]: Uploading segments... Aug 7 23:56:17.423847678 (14h16m17.013192665s) [upload=14789]: Got next segment. Inline: true Aug 7 23:56:17.423851335 (14h16m17.013196322s) [upload=14789]: Waiting for error group managing segments... Aug 7 23:56:17.423883377 (14h16m17.013228364s) [upload=14789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:17.423887054 (14h16m17.013232041s) [upload=14789]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:17.423889258 (14h16m17.013234245s) [upload=14789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:17.689327115 (14h16m17.278672102s) [upload=14789]: Done uploading segments... Aug 7 23:56:17.689345711 (14h16m17.278690688s) [upload=14789]: Upload finished. err: <nil> Aug 7 23:56:58.288741281 (14h16m57.878086268s) [upload=14801]: Starting upload Aug 7 23:56:58.288797098 (14h16m57.878142085s) [upload=14801]: Done starting upload Aug 7 23:56:58.288851010 (14h16m57.878195987s) [upload=14801]: Uploading segments... Aug 7 23:56:58.288882281 (14h16m57.878227268s) [upload=14801]: Got next segment. Inline: true Aug 7 23:56:58.288887911 (14h16m57.878232898s) [upload=14801]: Waiting for error group managing segments... Aug 7 23:56:58.288919792 (14h16m57.878264769s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:58.288923970 (14h16m57.878268957s) [upload=14801]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:58.288926485 (14h16m57.878271462s) [upload=14801]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:58.556529816 (14h16m58.145874803s) [upload=14801]: Done uploading segments... Aug 7 23:56:58.556544975 (14h16m58.145889962s) [upload=14801]: Upload finished. err: <nil> Aug 7 23:57:18.362794018 (14h17m17.952138995s) [upload=14806]: Starting upload Aug 7 23:57:18.362883679 (14h17m17.952228666s) [upload=14806]: Done starting upload Aug 7 23:57:18.362928104 (14h17m17.952273092s) [upload=14806]: Uploading segments... Aug 7 23:57:18.362957320 (14h17m17.952302307s) [upload=14806]: Got next segment. Inline: true Aug 7 23:57:18.362961077 (14h17m17.952306064s) [upload=14806]: Waiting for error group managing segments... Aug 7 23:57:18.363018407 (14h17m17.952363394s) [upload=14806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:18.363022404 (14h17m17.952367392s) [upload=14806]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:18.363024739 (14h17m17.952369716s) [upload=14806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:47.715297444 (14h17m47.304642431s) [upload=14806]: Done uploading segments... Aug 7 23:57:47.715410370 (14h17m47.304755357s) [upload=14806]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:53.375540080 (14h17m52.964885068s) [upload=14813]: Starting upload Aug 7 23:57:53.375600245 (14h17m52.964945232s) [upload=14813]: Done starting upload Aug 7 23:57:53.375654389 (14h17m52.964999376s) [upload=14813]: Uploading segments... Aug 7 23:57:53.375686622 (14h17m52.965031609s) [upload=14813]: Got next segment. Inline: true Aug 7 23:57:53.375690759 (14h17m52.965035746s) [upload=14813]: Waiting for error group managing segments... Aug 7 23:57:53.375725876 (14h17m52.965070863s) [upload=14813]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:53.375730134 (14h17m52.965075121s) [upload=14813]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:53.375734372 (14h17m52.965079359s) [upload=14813]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:22.726838464 (14h18m22.316183451s) [upload=14813]: Done uploading segments... Aug 7 23:58:22.726860075 (14h18m22.316205062s) [upload=14813]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.621870384 (14h18m23.211215371s) [upload=14822]: Starting upload Aug 7 23:58:23.621952000 (14h18m23.211296987s) [upload=14822]: Done starting upload Aug 7 23:58:23.622003359 (14h18m23.211348346s) [upload=14822]: Uploading segments... Aug 7 23:58:23.622043436 (14h18m23.211388413s) [upload=14822]: Got next segment. Inline: true Aug 7 23:58:23.622046772 (14h18m23.211391759s) [upload=14822]: Waiting for error group managing segments... Aug 7 23:58:23.622078573 (14h18m23.21142356s) [upload=14822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.622082230 (14h18m23.211427217s) [upload=14822]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.622084534 (14h18m23.211429511s) [upload=14822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:52.983677879 (14h18m52.573022876s) [upload=14822]: Done uploading segments... Aug 7 23:58:52.983713708 (14h18m52.573058685s) [upload=14822]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:11.446858470 (14h19m11.036203457s) [upload=14835]: Starting upload Aug 7 23:59:11.446906493 (14h19m11.03625148s) [upload=14835]: Done starting upload Aug 7 23:59:11.446947491 (14h19m11.036292479s) [upload=14835]: Uploading segments... Aug 7 23:59:11.446975475 (14h19m11.036320462s) [upload=14835]: Got next segment. Inline: true Aug 7 23:59:11.446978751 (14h19m11.036323738s) [upload=14835]: Waiting for error group managing segments... Aug 7 23:59:11.447008889 (14h19m11.036353876s) [upload=14835]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:11.447013638 (14h19m11.036358625s) [upload=14835]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:11.447016212 (14h19m11.0363612s) [upload=14835]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:40.799397635 (14h19m40.388742622s) [upload=14835]: Done uploading segments... Aug 7 23:59:40.799413266 (14h19m40.388758243s) [upload=14835]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:59.523080643 (14h19m59.11242563s) [upload=14847]: Starting upload Aug 7 23:59:59.523152089 (14h19m59.112497076s) [upload=14847]: Done starting upload Aug 7 23:59:59.523213727 (14h19m59.112558714s) [upload=14847]: Uploading segments... Aug 7 23:59:59.523251349 (14h19m59.112596336s) [upload=14847]: Got next segment. Inline: true Aug 7 23:59:59.523255287 (14h19m59.112600274s) [upload=14847]: Waiting for error group managing segments... Aug 7 23:59:59.523300013 (14h19m59.112645s) [upload=14847]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:59.523304251 (14h19m59.112649238s) [upload=14847]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:59.523307146 (14h19m59.112652133s) [upload=14847]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:28.877190143 (14h20m28.46653516s) [upload=14847]: Done uploading segments... Aug 8 00:00:28.877218477 (14h20m28.466563505s) [upload=14847]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.397064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X..............X........X..................] duration: 6.773945677s FAIL (2025-08-08 00:02:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 408ba8db7a0d8af05b7d5f8336d5c5f7f4f0063d72c98ffee3c421a5da299777 != before d0b3d19e255585f56d510f04d387098fcb83db6017871f75c2e11245b6143455 ======LOGS====== ished. err: <nil> Aug 7 23:48:24.457116068 (14h8m24.046461055s) [upload=14655]: Starting upload Aug 7 23:48:24.457179188 (14h8m24.046524175s) [upload=14655]: Done starting upload Aug 7 23:48:24.457293146 (14h8m24.046638133s) [upload=14655]: Uploading segments... Aug 7 23:48:24.457654307 (14h8m24.046999294s) [upload=14655]: Got next segment. Inline: false Aug 7 23:48:24.457709964 (14h8m24.047054951s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:24.457715013 (14h8m24.04706s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:48:24.901153105 (14h8m24.490498082s) [seg_pos={0 0}, upload=14655]: Begin upload segment... Aug 7 23:48:24.901189916 (14h8m24.490534903s) [seg_pos={0 0}, upload=14655]: Done begin upload segment. Aug 7 23:48:24.901208150 (14h8m24.490553137s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901547320 (14h8m24.490892297s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901866269 (14h8m24.491211257s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.901979486 (14h8m24.491324473s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902167115 (14h8m24.491512102s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902393378 (14h8m24.491738355s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902443794 (14h8m24.491788771s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902663996 (14h8m24.492008983s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.902931768 (14h8m24.492276755s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903278531 (14h8m24.492623518s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903535794 (14h8m24.492880781s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903745445 (14h8m24.493090432s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.903975075 (14h8m24.493320062s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904281360 (14h8m24.493626337s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904481643 (14h8m24.49382663s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904668631 (14h8m24.494013608s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.904852633 (14h8m24.49419762s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:24.905040943 (14h8m24.49438593s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Uploading piece... Aug 7 23:48:26.003752552 (14h8m25.593097539s) [upload=14655]: Next returned nil segment Aug 7 23:48:26.003765798 (14h8m25.593110785s) [upload=14655]: Waiting for error group managing segments... Aug 7 23:48:26.282577597 (14h8m25.871922574s) [noise=true , address=64.49.13.26:4951 , piece_id=YINSW2G4GU75CAT4WNX5RAWVHOQ4UKDCCM2LOPBQHHADYL3CICCQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.322820552 (14h8m25.912165529s) [noise=true , address=79.127.205.233:4642 , piece_id=QNW2JTOWEXPGSO2PPNRDACQCLLKKO3HOL7V5JNDGH2FPHSHGHBOA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.366928523 (14h8m25.95627351s) [noise=true , address=64.49.13.169:4713 , piece_id=DT5LD7HOM5BNHGMBAERVHZMO5HHA6WCEPRXFUKKEWGKQISUCDWTA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.822974095 (14h8m26.412319082s) [noise=true , address=104.36.82.18:4400 , piece_id=WPZKYGO2F2EDPLWCUYKEOO7G4AVRPPEMTXU4745P6CJ6NCN4DE5Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.895356812 (14h8m26.4847018s) [noise=true , address=79.127.201.215:4349 , piece_id=R4FXSPQNN5K4BJCVRV52KNC7N4ESRHE6A6SELY5C2IAGN5VSQWCA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.915282265 (14h8m26.504627252s) [noise=true , address=104.36.82.13:4212 , piece_id=JHHL2AQ27PTSCIJPU6RDTASKFWRN2WEWYRTZCJ2Y5R2JKQ5PGKBA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:26.929180326 (14h8m26.518525313s) [noise=true , address=104.36.82.62:4423 , piece_id=H4WAV63PP2TRCUN6ARA4UFQBLDOLZYLZJRXCBQYKDU2IY4HJPN6Q, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.126611753 (14h8m26.71595674s) [noise=true , address=45.91.116.5:4315 , piece_id=5MH7B4MI2OUABAEKZF2IIBB3AV4FA4HRRXUIRP3VGMTBJYCKIMVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.271660348 (14h8m26.861005335s) [noise=true , address=104.36.82.59:4303 , piece_id=Y3JYDT6A72UDDZAPWH6NWSHD2KZGM7C5SHZOYXARXDMZAHG3OTWQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.274412033 (14h8m26.86375702s) [noise=true , address=104.36.82.44:4888 , piece_id=DJ5E5LU4IBG2UECGR4MOZJXNHW7DHSIYVUI5HPX63UUQPAPUIDVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.336714535 (14h8m26.926059522s) [noise=true , address=45.91.116.13:4164 , piece_id=XAN7ADXGVGKWF2RIT7ZC2WEEDRCDI2FHKZEIE565YWW5HQGBLKTQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.759468572 (14h8m27.348813559s) [noise=true , address=104.36.82.37:4661 , piece_id=4JLQ56LWDYPBMUM7F2XEAJKHZT2CGUIK3SZXJ52ZEDCMNW35IQJQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.904353465 (14h8m27.493698452s) [noise=true , address=104.36.82.60:4371 , piece_id=N2KW2MYM5Y2HCG4VT437TB7QLR5LGHXPAVULR5IGGPGK5YRHGOHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960895364 (14h8m27.550240341s) [noise=true , address=104.36.82.37:4654 , piece_id=467RRANJMAQY5AJFUQIB4GO7MYCOKPGGXDNETNOEJ24P7X3YFTVA, seg_pos={0 0}, upload=14655]: Done uploading piece. err: <nil> Aug 7 23:48:27.960922416 (14h8m27.550267403s) [seg_pos={0 0}, upload=14655]: Segment reached optimal threshold of 14 pieces. Aug 7 23:48:27.961026686 (14h8m27.550371673s) [noise=true , address=104.36.82.16:4299 , piece_id=LUTQQPCUC3YITHOIW2G2JLN74ET77L5XJZXMFGRNU63SY33MOIOQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961266504 (14h8m27.550611491s) [noise=true , address=104.36.82.3:4785 , piece_id=BPL7HHDZMMPTCUYK5JS67JXGOKK3SCS36UXJ3EL24YGPCYIBNG5A, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9): protocol: expected piece hash; context canceled; EOF Aug 7 23:48:27.961284479 (14h8m27.550629456s) [noise=true , address=104.36.82.5:4869 , piece_id=EVCBVWD7YB6UOF2ZK5WLIVYOQ6474PYTMXC2USREW6AAU67CXRSQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr): protocol: rpc: dial tcp 104.36.82.5:4869: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:48:27.961415680 (14h8m27.550760667s) [noise=true , address=104.36.82.47:4541 , piece_id=WMCMET3A3L6PE5OVONBTCIB4SYLRVIQINWKX7M27SK7XAQV5ILHQ, seg_pos={0 0}, upload=14655]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC): protocol: context canceled; context canceled Aug 7 23:48:27.961429706 (14h8m27.550774693s) [seg_pos={0 0}, upload=14655]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:48:27.961507495 (14h8m27.550852482s) [upload=14655]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:48:27.961511613 (14h8m27.5508566s) [upload=14655]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:28.311226257 (14h8m27.900571244s) [upload=14655]: Done uploading segments... Aug 7 23:48:28.311246106 (14h8m27.900591083s) [upload=14655]: Upload finished. err: <nil> Duration: 6.774009599s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.X.....X.X...................................X.........X.X..] duration: 2.293544424s FAIL (2025-08-08 00:02:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.293587177s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X............................................XX..XXXX....XX.] duration: 6.579569581s FAIL (2025-08-08 00:02:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.700898256s) [upload=14855]: Done starting upload Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments... Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments... Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments... Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment... Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment. Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments... Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces. Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments... Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.579588746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X............................................XXX.XX....] duration: 5.941927369s FAIL (2025-08-08 00:02:01 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.941960282s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXX.X..XX.......................................X....X......] duration: 6.22804281s FAIL (2025-08-08 00:02:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.228068779s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X............................................XX..XXXX....XX] duration: 38.835346474s FAIL (2025-08-08 00:01:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3.700898256s) [upload=14855]: Done starting upload Aug 8 00:00:34.111602513 (14h20m33.7009475s) [upload=14855]: Uploading segments... Aug 8 00:00:34.111774201 (14h20m33.701119218s) [upload=14855]: Got next segment. Inline: false Aug 8 00:00:34.111816783 (14h20m33.70116178s) [upload=14855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.111822464 (14h20m33.701167461s) [upload=14855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:03.465979471 (14h21m3.055324468s) [upload=14855]: Next segment err: context deadline exceeded Aug 8 00:01:03.466148254 (14h21m3.055493251s) [upload=14855]: Done uploading segments... Aug 8 00:01:03.466174945 (14h21m3.055519932s) [upload=14855]: Upload finished. err: context deadline exceeded Aug 8 00:01:19.453976971 (14h21m19.043321958s) [upload=14872]: Starting upload Aug 8 00:01:19.454042106 (14h21m19.043387123s) [upload=14872]: Done starting upload Aug 8 00:01:19.454156104 (14h21m19.043501091s) [upload=14872]: Uploading segments... Aug 8 00:01:19.454244503 (14h21m19.04358949s) [upload=14872]: Got next segment. Inline: false Aug 8 00:01:19.454272567 (14h21m19.043617554s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:19.454276825 (14h21m19.043621812s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:01:39.582519114 (14h21m39.171864101s) [seg_pos={0 0}, upload=14872]: Begin upload segment... Aug 8 00:01:39.582565882 (14h21m39.17191087s) [seg_pos={0 0}, upload=14872]: Done begin upload segment. Aug 8 00:01:39.582581332 (14h21m39.17192632s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582590870 (14h21m39.171935857s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.582951871 (14h21m39.172296858s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583059888 (14h21m39.172404905s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583214905 (14h21m39.172559892s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583427882 (14h21m39.172772869s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583689032 (14h21m39.173034019s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.583925344 (14h21m39.173270331s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584272317 (14h21m39.173617294s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584579135 (14h21m39.173924122s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.584852117 (14h21m39.174197104s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585105202 (14h21m39.174450179s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585298762 (14h21m39.174643739s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585506369 (14h21m39.174851346s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.585763882 (14h21m39.175108869s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:39.586021575 (14h21m39.175366562s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Uploading piece... Aug 8 00:01:41.370103161 (14h21m40.959448138s) [upload=14872]: Next returned nil segment Aug 8 00:01:41.370130243 (14h21m40.95947523s) [upload=14872]: Waiting for error group managing segments... Aug 8 00:01:41.673384945 (14h21m41.262729922s) [noise=true , address=104.36.82.2:4731 , piece_id=2BTN7TE7VVSPN5BUCDVKHWZPPATMNIJLXNDXUO46Z7CRB3QBCUEQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.739890780 (14h21m41.329235767s) [noise=true , address=104.36.82.9:4005 , piece_id=ZYZ6VK5MCW7QE6N2GOYW2WMWDEWGONNK2Y7QBE7FPVPLY2OZGIMQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.743845828 (14h21m41.333190815s) [noise=true , address=104.36.82.8:4970 , piece_id=N52DS4AB7Z6VSFYWHN5MXPUS3AP7UKN2K7U5D5JZWETM7ZLROO7Q, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.786086095 (14h21m41.375431072s) [noise=true , address=45.91.116.8:4047 , piece_id=4LUGFRA2QPUK5LPKZGNTURXXSGY64AZU4Q3BHBAM3SPBZ7M2FFKQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.791525722 (14h21m41.380870699s) [noise=true , address=104.36.82.5:4839 , piece_id=EHHLJG3INARPORGO6YVXU7QJAK23ADRVWBGNT5IKL4N6Y4ASM3XA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:41.959657137 (14h21m41.549002124s) [noise=true , address=104.36.82.13:4185 , piece_id=O26DWLA45AEOUZOW2CZTLJNPVRWB5KVDQW6FH2ANTKLUMYNWRVKA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.000743686 (14h21m41.590088693s) [noise=true , address=45.91.116.12:4225 , piece_id=MUZ4UWUJSWGYKIO3BSTEOJYBBXSK4R2PURNDXHOYLTZ7VZ466IGA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.160166408 (14h21m41.749511395s) [noise=true , address=45.91.116.4:4940 , piece_id=5S5JLBIGSBIVTPVJT5MIRAE42PRCIFWDNWLY7NQXSWPMHT5WSTRA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175399653 (14h21m41.764744641s) [noise=true , address=45.91.116.14:4190 , piece_id=FSO3UN2D24KPDVSWWPI3ROBXN7RN6D2HAZKOXEQ2JOW2M6RQ5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.175880162 (14h21m41.765225139s) [noise=true , address=45.91.116.5:4299 , piece_id=HRKD77XA4SHZKSEW2WA3AROWPLHOWEXG5GEOWJCG7NTFOYYZ4DFA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.278394081 (14h21m41.867739068s) [noise=true , address=77.90.31.21:4877 , piece_id=MZ4WJDOFWH56JIAGXJZ7FOAPMNCHVEBUUM2G5IY2KKPRHPJH5MDA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.293663936 (14h21m41.883008933s) [noise=true , address=77.90.31.18:4807 , piece_id=V7EZJW5PYJCWWNNIRAJCFTT42DQJY2C4RDHB5645TKOSXJJXE2WA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309113927 (14h21m41.898458904s) [noise=true , address=45.91.116.9:4263 , piece_id=F4QDS3EOSLFFCEDU6N56AMMQR4GHPWZ6TIAMO5SGVIMVMFA47EQQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: <nil> Aug 8 00:01:42.309137942 (14h21m41.898482919s) [seg_pos={0 0}, upload=14872]: Segment reached optimal threshold of 13 pieces. Aug 8 00:01:42.309315973 (14h21m41.89866099s) [noise=true , address=77.90.31.11:4555 , piece_id=OCZBWUJHEVLMUD6BNADJ7BOSHHZPESQJOSKDZ3T2MUSQRFTNI7KA, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309402328 (14h21m41.898747305s) [noise=true , address=104.36.82.10:4079 , piece_id=262SP4DREL7FYDG6YOBFWDPWKYHZIQGDJKSX7XAIUJUTQWUIEEZQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Aug 8 00:01:42.309425222 (14h21m41.898770229s) [noise=true , address=104.36.82.11:4115 , piece_id=S34RF36SERFPPSHBPYJ5V3FCR5ZE7RFIVEXBOTFRHPEEIUCHO2LQ, seg_pos={0 0}, upload=14872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42778->104.36.82.11:4115: use of closed network connection Aug 8 00:01:42.309450360 (14h21m41.898795347s) [seg_pos={0 0}, upload=14872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 8 00:01:42.309540663 (14h21m41.89888565s) [upload=14872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:01:42.309548688 (14h21m41.898893735s) [upload=14872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:57.644545293 (14h21m57.2338903s) [upload=14872]: Done uploading segments... Aug 8 00:01:57.644581281 (14h21m57.233926288s) [upload=14872]: Upload finished. err: metaclient: context deadline exceeded Duration: 38.83537517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X................................X.........................] duration: 2.408238286s FAIL (2025-08-08 00:01:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== pload=14406]: Done uploading segments... Aug 7 23:33:49.820540362 (13h53m49.409885379s) [upload=14406]: Upload finished. err: <nil> Aug 7 23:34:55.875843674 (13h54m55.465188661s) [upload=14425]: Starting upload Aug 7 23:34:55.875888460 (13h54m55.465233447s) [upload=14425]: Done starting upload Aug 7 23:34:55.875931803 (13h54m55.46527678s) [upload=14425]: Uploading segments... Aug 7 23:34:55.875961150 (13h54m55.465306137s) [upload=14425]: Got next segment. Inline: true Aug 7 23:34:55.875964626 (13h54m55.465309603s) [upload=14425]: Waiting for error group managing segments... Aug 7 23:34:55.875996888 (13h54m55.465341875s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:34:55.876001015 (13h54m55.465346002s) [upload=14425]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:34:55.876003680 (13h54m55.465348667s) [upload=14425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:34:56.158049155 (13h54m55.747394142s) [upload=14425]: Done uploading segments... Aug 7 23:34:56.158066439 (13h54m55.747411426s) [upload=14425]: Upload finished. err: <nil> Aug 7 23:35:36.771198141 (13h55m36.360543128s) [upload=14436]: Starting upload Aug 7 23:35:36.771239510 (13h55m36.360584487s) [upload=14436]: Done starting upload Aug 7 23:35:36.771282783 (13h55m36.36062776s) [upload=14436]: Uploading segments... Aug 7 23:35:36.771311258 (13h55m36.360656245s) [upload=14436]: Got next segment. Inline: true Aug 7 23:35:36.771315045 (13h55m36.360660032s) [upload=14436]: Waiting for error group managing segments... Aug 7 23:35:36.771344952 (13h55m36.360689939s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:35:36.771350051 (13h55m36.360695038s) [upload=14436]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:35:36.771352185 (13h55m36.360697172s) [upload=14436]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:35:37.066875398 (13h55m36.656220385s) [upload=14436]: Done uploading segments... Aug 7 23:35:37.066894275 (13h55m36.656239262s) [upload=14436]: Upload finished. err: <nil> Aug 7 23:36:11.909385420 (13h56m11.498730407s) [upload=14444]: Starting upload Aug 7 23:36:11.909445175 (13h56m11.498790172s) [upload=14444]: Done starting upload Aug 7 23:36:11.909503978 (13h56m11.498848985s) [upload=14444]: Uploading segments... Aug 7 23:36:11.909562349 (13h56m11.498907356s) [upload=14444]: Got next segment. Inline: true Aug 7 23:36:11.909568220 (13h56m11.498913207s) [upload=14444]: Waiting for error group managing segments... Aug 7 23:36:11.909608778 (13h56m11.498953765s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:11.909614439 (13h56m11.498959416s) [upload=14444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:11.909617855 (13h56m11.498962842s) [upload=14444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:12.226741776 (13h56m11.816086753s) [upload=14444]: Done uploading segments... Aug 7 23:36:12.226761765 (13h56m11.816106752s) [upload=14444]: Upload finished. err: <nil> Aug 7 23:36:45.061365955 (13h56m44.650710942s) [upload=14452]: Starting upload Aug 7 23:36:45.061414287 (13h56m44.650759274s) [upload=14452]: Done starting upload Aug 7 23:36:45.061459804 (13h56m44.650804792s) [upload=14452]: Uploading segments... Aug 7 23:36:45.061489772 (13h56m44.650834759s) [upload=14452]: Got next segment. Inline: true Aug 7 23:36:45.061493318 (13h56m44.650838305s) [upload=14452]: Waiting for error group managing segments... Aug 7 23:36:45.061525611 (13h56m44.650870598s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:36:45.061529408 (13h56m44.650874395s) [upload=14452]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:36:45.061531542 (13h56m44.650876529s) [upload=14452]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:36:45.340854790 (13h56m44.930199777s) [upload=14452]: Done uploading segments... Aug 7 23:36:45.340871481 (13h56m44.930216468s) [upload=14452]: Upload finished. err: <nil> Aug 7 23:37:27.165030871 (13h57m26.754375858s) [upload=14466]: Starting upload Aug 7 23:37:27.165168204 (13h57m26.754513201s) [upload=14466]: Done starting upload Aug 7 23:37:27.165252034 (13h57m26.754597061s) [upload=14466]: Uploading segments... Aug 7 23:37:27.165304224 (13h57m26.754649251s) [upload=14466]: Got next segment. Inline: true Aug 7 23:37:27.165311979 (13h57m26.754657006s) [upload=14466]: Waiting for error group managing segments... Aug 7 23:37:27.165356895 (13h57m26.754701872s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:37:27.165361664 (13h57m26.754706651s) [upload=14466]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:37:27.165364449 (13h57m26.754709426s) [upload=14466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:37:27.453150839 (13h57m27.042495816s) [upload=14466]: Done uploading segments... Aug 7 23:37:27.453172210 (13h57m27.042517218s) [upload=14466]: Upload finished. err: <nil> Aug 7 23:38:33.292631280 (13h58m32.881976267s) [upload=14482]: Starting upload Aug 7 23:38:33.292682438 (13h58m32.882027425s) [upload=14482]: Done starting upload Aug 7 23:38:33.292735750 (13h58m32.882080737s) [upload=14482]: Uploading segments... Aug 7 23:38:33.292771078 (13h58m32.882116065s) [upload=14482]: Got next segment. Inline: true Aug 7 23:38:33.292774524 (13h58m32.882119501s) [upload=14482]: Waiting for error group managing segments... Aug 7 23:38:33.292809602 (13h58m32.882154589s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:33.292814000 (13h58m32.882158987s) [upload=14482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:33.292816806 (13h58m32.882161793s) [upload=14482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:33.652660690 (13h58m33.242005677s) [upload=14482]: Done uploading segments... Aug 7 23:38:33.652677502 (13h58m33.242022489s) [upload=14482]: Upload finished. err: <nil> Aug 7 23:38:56.173816507 (13h58m55.763161484s) [upload=14490]: Starting upload Aug 7 23:38:56.173867385 (13h58m55.763212372s) [upload=14490]: Done starting upload Aug 7 23:38:56.173920446 (13h58m55.763265433s) [upload=14490]: Uploading segments... Aug 7 23:38:56.173947418 (13h58m55.763292405s) [upload=14490]: Got next segment. Inline: true Aug 7 23:38:56.173951646 (13h58m55.763296633s) [upload=14490]: Waiting for error group managing segments... Aug 7 23:38:56.173986452 (13h58m55.763331439s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:38:56.173990099 (13h58m55.763335086s) [upload=14490]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:38:56.173992515 (13h58m55.763337492s) [upload=14490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:38:56.442927417 (13h58m56.032272424s) [upload=14490]: Done uploading segments... Aug 7 23:38:56.442950000 (13h58m56.032294987s) [upload=14490]: Upload finished. err: <nil> Aug 7 23:39:51.382592315 (13h59m50.971937292s) [upload=14506]: Starting upload Aug 7 23:39:51.382637612 (13h59m50.971982599s) [upload=14506]: Done starting upload Aug 7 23:39:51.382703347 (13h59m50.972048324s) [upload=14506]: Uploading segments... Aug 7 23:39:51.382762962 (13h59m50.972107949s) [upload=14506]: Got next segment. Inline: true Aug 7 23:39:51.382767260 (13h59m50.972112247s) [upload=14506]: Waiting for error group managing segments... Aug 7 23:39:51.382800283 (13h59m50.97214527s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:39:51.382803920 (13h59m50.972148907s) [upload=14506]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:39:51.382806144 (13h59m50.972151131s) [upload=14506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:39:51.664212669 (13h59m51.253557746s) [upload=14506]: Done uploading segments... Aug 7 23:39:51.664238148 (13h59m51.253583125s) [upload=14506]: Upload finished. err: <nil> Aug 7 23:40:08.314581416 (14h0m7.903926403s) [upload=14509]: Starting upload Aug 7 23:40:08.314628847 (14h0m7.903973834s) [upload=14509]: Done starting upload Aug 7 23:40:08.314675916 (14h0m7.904020903s) [upload=14509]: Uploading segments... Aug 7 23:40:08.314709010 (14h0m7.904053998s) [upload=14509]: Got next segment. Inline: true Aug 7 23:40:08.314712467 (14h0m7.904057454s) [upload=14509]: Waiting for error group managing segments... Aug 7 23:40:08.314744288 (14h0m7.904089275s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:40:08.314748175 (14h0m7.904093162s) [upload=14509]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:40:08.314750489 (14h0m7.904095476s) [upload=14509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:40:08.646371868 (14h0m8.235716855s) [upload=14509]: Done uploading segments... Aug 7 23:40:08.646389852 (14h0m8.235734829s) [upload=14509]: Upload finished. err: <nil> Aug 7 23:41:18.965192405 (14h1m18.554537392s) [upload=14528]: Starting upload Aug 7 23:41:18.965250195 (14h1m18.554595172s) [upload=14528]: Done starting upload Aug 7 23:41:18.965313206 (14h1m18.554658193s) [upload=14528]: Uploading segments... Aug 7 23:41:18.965345408 (14h1m18.554690395s) [upload=14528]: Got next segment. Inline: true Aug 7 23:41:18.965349346 (14h1m18.554694333s) [upload=14528]: Waiting for error group managing segments... Aug 7 23:41:18.965382669 (14h1m18.554727656s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:41:18.965386546 (14h1m18.554731523s) [upload=14528]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:41:18.965389191 (14h1m18.554734178s) [upload=14528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:41:19.261666757 (14h1m18.851011744s) [upload=14528]: Done uploading segments... Aug 7 23:41:19.261683329 (14h1m18.851028316s) [upload=14528]: Upload finished. err: <nil> Duration: 2.408259747s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 9 out of 60 is failed (threshold=3) [XXXX..............................................X.XXXX....] duration: 1.887551713s FAIL (2025-08-08 00:01:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 83 (14h9m4.18419687s) [upload=14666]: Starting upload Aug 7 23:49:04.594900156 (14h9m4.184245143s) [upload=14666]: Done starting upload Aug 7 23:49:04.594944311 (14h9m4.184289298s) [upload=14666]: Uploading segments... Aug 7 23:49:04.594984397 (14h9m4.184329384s) [upload=14666]: Got next segment. Inline: true Aug 7 23:49:04.594987814 (14h9m4.184332801s) [upload=14666]: Waiting for error group managing segments... Aug 7 23:49:04.595016589 (14h9m4.184361576s) [upload=14666]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:04.595020496 (14h9m4.184365473s) [upload=14666]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:04.595022851 (14h9m4.184367828s) [upload=14666]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:04.868476128 (14h9m4.457821115s) [upload=14666]: Done uploading segments... Aug 7 23:49:04.868494303 (14h9m4.457839291s) [upload=14666]: Upload finished. err: <nil> Aug 7 23:50:10.977644727 (14h10m10.566989704s) [upload=14685]: Starting upload Aug 7 23:50:10.977704220 (14h10m10.567049207s) [upload=14685]: Done starting upload Aug 7 23:50:10.977750239 (14h10m10.567095226s) [upload=14685]: Uploading segments... Aug 7 23:50:10.977820534 (14h10m10.567165531s) [upload=14685]: Got next segment. Inline: true Aug 7 23:50:10.977824882 (14h10m10.567169869s) [upload=14685]: Waiting for error group managing segments... Aug 7 23:50:10.977858386 (14h10m10.567203373s) [upload=14685]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:10.977862343 (14h10m10.56720732s) [upload=14685]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:10.977864667 (14h10m10.567209644s) [upload=14685]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:11.274547609 (14h10m10.863892596s) [upload=14685]: Done uploading segments... Aug 7 23:50:11.274564251 (14h10m10.863909238s) [upload=14685]: Upload finished. err: <nil> Aug 7 23:50:53.129746032 (14h10m52.719091019s) [upload=14698]: Starting upload Aug 7 23:50:53.129817549 (14h10m52.719162526s) [upload=14698]: Done starting upload Aug 7 23:50:53.129869308 (14h10m52.719214285s) [upload=14698]: Uploading segments... Aug 7 23:50:53.129899716 (14h10m52.719244703s) [upload=14698]: Got next segment. Inline: true Aug 7 23:50:53.129903483 (14h10m52.719248471s) [upload=14698]: Waiting for error group managing segments... Aug 7 23:50:53.129935625 (14h10m52.719280612s) [upload=14698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:53.129939452 (14h10m52.719284439s) [upload=14698]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:53.129941756 (14h10m52.719286743s) [upload=14698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:53.612932053 (14h10m53.20227704s) [upload=14698]: Done uploading segments... Aug 7 23:50:53.612949836 (14h10m53.202294823s) [upload=14698]: Upload finished. err: <nil> Aug 7 23:51:55.415342045 (14h11m55.004687042s) [upload=14719]: Starting upload Aug 7 23:51:55.415462836 (14h11m55.004807824s) [upload=14719]: Done starting upload Aug 7 23:51:55.415539092 (14h11m55.004884089s) [upload=14719]: Uploading segments... Aug 7 23:51:55.415600830 (14h11m55.004945858s) [upload=14719]: Got next segment. Inline: true Aug 7 23:51:55.415607002 (14h11m55.004951989s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:51:55.415668760 (14h11m55.005013747s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:55.415675372 (14h11m55.005020379s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:55.415679240 (14h11m55.005024237s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:55.825969401 (14h11m55.415314378s) [upload=14719]: Done uploading segments... Aug 7 23:51:55.825985652 (14h11m55.415330629s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:52:50.979559156 (14h12m50.568904143s) [upload=14734]: Starting upload Aug 7 23:52:50.979632076 (14h12m50.568977063s) [upload=14734]: Done starting upload Aug 7 23:52:50.979699916 (14h12m50.569044903s) [upload=14734]: Uploading segments... Aug 7 23:52:50.979747066 (14h12m50.569092053s) [upload=14734]: Got next segment. Inline: true Aug 7 23:52:50.979751975 (14h12m50.569096962s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:52:50.979789868 (14h12m50.569134845s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:50.979794427 (14h12m50.569139414s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:50.979796861 (14h12m50.569141848s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:50.328281597 (14h13m49.917626584s) [upload=14734]: Done uploading segments... Aug 7 23:53:50.328316053 (14h13m49.91766103s) [upload=14734]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:04.489764266 (14h14m4.079109253s) [upload=14753]: Starting upload Aug 7 23:54:04.489828037 (14h14m4.079173014s) [upload=14753]: Done starting upload Aug 7 23:54:04.489877612 (14h14m4.079222589s) [upload=14753]: Uploading segments... Aug 7 23:54:04.489903933 (14h14m4.07924892s) [upload=14753]: Got next segment. Inline: true Aug 7 23:54:04.489907469 (14h14m4.079252446s) [upload=14753]: Waiting for error group managing segments... Aug 7 23:54:04.489939651 (14h14m4.079284638s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:04.489943298 (14h14m4.079288285s) [upload=14753]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:04.489945472 (14h14m4.079290459s) [upload=14753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:04.778726062 (14h14m4.368071049s) [upload=14753]: Done uploading segments... Aug 7 23:54:04.778744989 (14h14m4.368089976s) [upload=14753]: Upload finished. err: <nil> Aug 7 23:55:14.199719855 (14h15m13.789064892s) [upload=14770]: Starting upload Aug 7 23:55:14.199777645 (14h15m13.789122632s) [upload=14770]: Done starting upload Aug 7 23:55:14.199824415 (14h15m13.789169392s) [upload=14770]: Uploading segments... Aug 7 23:55:14.199852819 (14h15m13.789197806s) [upload=14770]: Got next segment. Inline: true Aug 7 23:55:14.199856436 (14h15m13.789201423s) [upload=14770]: Waiting for error group managing segments... Aug 7 23:55:14.199889419 (14h15m13.789234406s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:14.199893557 (14h15m13.789238544s) [upload=14770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:14.199895951 (14h15m13.789240938s) [upload=14770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:13.542516296 (14h16m13.131861283s) [upload=14770]: Done uploading segments... Aug 7 23:56:13.542536334 (14h16m13.131881321s) [upload=14770]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:14.428677736 (14h16m14.018022723s) [upload=14788]: Starting upload Aug 7 23:56:14.428730978 (14h16m14.018075965s) [upload=14788]: Done starting upload Aug 7 23:56:14.428789289 (14h16m14.018134276s) [upload=14788]: Uploading segments... Aug 7 23:56:14.428819026 (14h16m14.018164013s) [upload=14788]: Got next segment. Inline: true Aug 7 23:56:14.428822873 (14h16m14.01816785s) [upload=14788]: Waiting for error group managing segments... Aug 7 23:56:14.428855215 (14h16m14.018200202s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:14.428858562 (14h16m14.018203549s) [upload=14788]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:14.428860736 (14h16m14.018205723s) [upload=14788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:13.781984834 (14h17m13.371329821s) [upload=14788]: Done uploading segments... Aug 7 23:57:13.782003370 (14h17m13.371348357s) [upload=14788]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:29.079359478 (14h17m28.668704466s) [upload=14808]: Starting upload Aug 7 23:57:29.079416147 (14h17m28.668761134s) [upload=14808]: Done starting upload Aug 7 23:57:29.079461845 (14h17m28.668806822s) [upload=14808]: Uploading segments... Aug 7 23:57:29.079488876 (14h17m28.668833863s) [upload=14808]: Got next segment. Inline: true Aug 7 23:57:29.079492263 (14h17m28.66883725s) [upload=14808]: Waiting for error group managing segments... Aug 7 23:57:29.079523392 (14h17m28.668868379s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:29.079527049 (14h17m28.668872036s) [upload=14808]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:29.079529484 (14h17m28.668874461s) [upload=14808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:28.439377526 (14h18m28.028722513s) [upload=14808]: Done uploading segments... Aug 7 23:58:28.439395721 (14h18m28.028740708s) [upload=14808]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:29.333669502 (14h18m28.923014489s) [upload=14825]: Starting upload Aug 7 23:58:29.333742581 (14h18m28.923087568s) [upload=14825]: Done starting upload Aug 7 23:58:29.333835740 (14h18m28.923180727s) [upload=14825]: Uploading segments... Aug 7 23:58:29.333878000 (14h18m28.923222987s) [upload=14825]: Got next segment. Inline: true Aug 7 23:58:29.333881998 (14h18m28.923226975s) [upload=14825]: Waiting for error group managing segments... Aug 7 23:58:29.333911455 (14h18m28.923256442s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:29.333917496 (14h18m28.923262473s) [upload=14825]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:29.333919741 (14h18m28.923264728s) [upload=14825]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:28.692415040 (14h19m28.281760027s) [upload=14825]: Done uploading segments... Aug 7 23:59:28.692444777 (14h19m28.281789764s) [upload=14825]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.887576369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.X.X............................................XX..X.......] duration: 1.915415686s FAIL (2025-08-08 00:01:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.915441757s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 645.660693ms target: https://us1.storj.io/login FAIL (2025-08-08 00:01:46 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 645.683876ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXX.X.X.........................................XXXXXX...] duration: 1m0.001478781s FAIL (2025-08-08 00:01:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001495082s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)