Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X.] duration: 7.283177672s trace: https://jaeger.dev.storj.io/trace/5889cd461255aa32 FAIL (2025-12-26 17:14:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:13:19.852963960 (505h44m19.724047005s) [upload=443784]: Starting upload Dec 26 17:13:19.853054533 (505h44m19.724137608s) [upload=443784]: Done starting upload Dec 26 17:13:19.853139835 (505h44m19.72422288s) [upload=443784]: Uploading segments... Dec 26 17:13:19.853377808 (505h44m19.724460843s) [upload=443784]: Got next segment. Inline: false Dec 26 17:13:19.853419167 (505h44m19.724502232s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:13:19.853425068 (505h44m19.724508103s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:13:20.097289779 (505h44m19.968372824s) [seg_pos={0 0}, upload=443784]: Begin upload segment... Dec 26 17:13:20.097342379 (505h44m19.968425414s) [seg_pos={0 0}, upload=443784]: Done begin upload segment. Dec 26 17:13:20.097359672 (505h44m19.968442717s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097371083 (505h44m19.968454118s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097454262 (505h44m19.968537297s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098233846 (505h44m19.969316881s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098533247 (505h44m19.969616282s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098840783 (505h44m19.969923828s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099108271 (505h44m19.970191306s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099390849 (505h44m19.970473884s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099739633 (505h44m19.970822668s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100089951 (505h44m19.971172986s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100892640 (505h44m19.971975675s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101061561 (505h44m19.972144596s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101987524 (505h44m19.973070559s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.102679572 (505h44m19.973762607s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103054747 (505h44m19.974137782s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103929292 (505h44m19.975012337s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:21.814556191 (505h44m21.685639226s) [upload=443784]: Next returned nil segment Dec 26 17:13:21.814589354 (505h44m21.685672389s) [upload=443784]: Waiting for error group managing segments... Dec 26 17:13:22.104277310 (505h44m21.975360355s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.108309108 (505h44m21.979392153s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.110933244 (505h44m21.982016289s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111461040 (505h44m21.982544075s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111506646 (505h44m21.982589792s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.113182127 (505h44m21.984265172s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.122792306 (505h44m21.993875341s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124506099 (505h44m21.995589144s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124710539 (505h44m21.995793584s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.133759890 (505h44m22.004842935s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.505610818 (505h44m22.376693864s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.629039548 (505h44m22.500122593s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641837758 (505h44m22.512920803s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641862726 (505h44m22.512945761s) [seg_pos={0 0}, upload=443784]: Segment reached optimal threshold of 13 pieces. Dec 26 17:13:22.642195420 (505h44m22.513278465s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642203645 (505h44m22.51328669s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642301421 (505h44m22.513384466s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642311620 (505h44m22.513394656s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642589900 (505h44m22.513672955s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: context canceled; context canceled Dec 26 17:13:22.642599058 (505h44m22.513682093s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642620228 (505h44m22.513703273s) [seg_pos={0 0}, upload=443784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:13:22.642698367 (505h44m22.513781412s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:13:22.642705129 (505h44m22.513788175s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:13:22.932721441 (505h44m22.803804476s) [upload=443784]: Done uploading segments... Dec 26 17:13:22.932740437 (505h44m22.803823482s) [upload=443784]: Upload finished. err: <nil> Duration: 7.283205083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......X....X................................] duration: 37.656463194s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:14:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 37.656481339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X......XXX...X........X...................................] duration: 3.515350922s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:13:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515427417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..X] duration: 30.001930678s trace: https://jaeger.dev.storj.io/trace/5889cd461255aa32 FAIL (2025-12-26 17:13:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:13:19.852963960 (505h44m19.724047005s) [upload=443784]: Starting upload Dec 26 17:13:19.853054533 (505h44m19.724137608s) [upload=443784]: Done starting upload Dec 26 17:13:19.853139835 (505h44m19.72422288s) [upload=443784]: Uploading segments... Dec 26 17:13:19.853377808 (505h44m19.724460843s) [upload=443784]: Got next segment. Inline: false Dec 26 17:13:19.853419167 (505h44m19.724502232s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:13:19.853425068 (505h44m19.724508103s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:13:20.097289779 (505h44m19.968372824s) [seg_pos={0 0}, upload=443784]: Begin upload segment... Dec 26 17:13:20.097342379 (505h44m19.968425414s) [seg_pos={0 0}, upload=443784]: Done begin upload segment. Dec 26 17:13:20.097359672 (505h44m19.968442717s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097371083 (505h44m19.968454118s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.097454262 (505h44m19.968537297s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098233846 (505h44m19.969316881s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098533247 (505h44m19.969616282s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.098840783 (505h44m19.969923828s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099108271 (505h44m19.970191306s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099390849 (505h44m19.970473884s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.099739633 (505h44m19.970822668s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100089951 (505h44m19.971172986s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.100892640 (505h44m19.971975675s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101061561 (505h44m19.972144596s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.101987524 (505h44m19.973070559s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.102679572 (505h44m19.973762607s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103054747 (505h44m19.974137782s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:20.103929292 (505h44m19.975012337s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Uploading piece... Dec 26 17:13:21.814556191 (505h44m21.685639226s) [upload=443784]: Next returned nil segment Dec 26 17:13:21.814589354 (505h44m21.685672389s) [upload=443784]: Waiting for error group managing segments... Dec 26 17:13:22.104277310 (505h44m21.975360355s) [noise=true , address=145.79.227.52:4447 , piece_id=VGQUFWVK3WFQ6DYJLCHYG5B23JTQBJ72HTQ4JMMJLFUVKNDY6SXA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.108309108 (505h44m21.979392153s) [noise=true , address=145.79.227.49:4338 , piece_id=JCF45FUL4DRKMNAL6VUQZQJIMMLL3AERGGCPNIFFNY6UPXOEE4YQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.110933244 (505h44m21.982016289s) [noise=true , address=145.79.227.8:4330 , piece_id=L3RXLGQS6MRZ523B2UCQY2QVA6PPJZWI3VXSHBN7PVTHVSML4PIQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111461040 (505h44m21.982544075s) [noise=true , address=145.79.227.41:4085 , piece_id=SR6B3PSWKZBXS2O7KUKJPASOR7PM24MRKYNTBQMF2TSYZH5Y6FVA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.111506646 (505h44m21.982589792s) [noise=true , address=145.79.227.45:4231 , piece_id=UC4ERWDUZUAWGVTI2KWGSNFSZ4KDWQWJPWDUWDJJALYHPRUQBCRA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.113182127 (505h44m21.984265172s) [noise=true , address=145.79.227.11:4731 , piece_id=QV4GJNUQ7ANXVGTXW3YDV4VOY5WZMGDTU62ETPATS4NYPBDWX7IA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.122792306 (505h44m21.993875341s) [noise=true , address=145.79.227.31:4799 , piece_id=OLS4J6V7U5UQ2SXTVRSWY3VV76ST74HP77NEG6DEAUVIZISSTHGA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124506099 (505h44m21.995589144s) [noise=true , address=145.79.227.43:4159 , piece_id=Q7AFUYAZO66BSKEH7TUONLI5HQDWD5J6GAPPWNCY26BVTJA6RCJQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.124710539 (505h44m21.995793584s) [noise=true , address=145.79.227.36:4517 , piece_id=ODC43SDUH2ASENQ4JV2JIKDGTHU3MW3DM65ABM2C3RQLQRPILC7Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.133759890 (505h44m22.004842935s) [noise=true , address=145.79.227.22:4511 , piece_id=UJJXSCRGFTGXRYCQIW35YLZOSL7HCQB7MOYA5PNDDUSD34GPAZ4A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.505610818 (505h44m22.376693864s) [noise=true , address=77.90.31.14:4661 , piece_id=DXV7SFOQICU5LM7AHYO5QEZJB5TDJPABNCTC2YKHLNQWDOYDSCTA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.629039548 (505h44m22.500122593s) [noise=true , address=77.90.31.21:4878 , piece_id=UAPCOPFEXAAJCF2SKEFJXU2Z55JHJ5NZLCVMMCSVP4KD226OD63Q, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641837758 (505h44m22.512920803s) [noise=true , address=77.90.31.10:4517 , piece_id=DC3WXFHG2JZMB4KG64BRFH253PTY455P7RL5KIBLFKTGOPBPN73A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: <nil> Dec 26 17:13:22.641862726 (505h44m22.512945761s) [seg_pos={0 0}, upload=443784]: Segment reached optimal threshold of 13 pieces. Dec 26 17:13:22.642195420 (505h44m22.513278465s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642203645 (505h44m22.51328669s) [noise=true , address=45.91.116.10:4083 , piece_id=UCQURKWRA6YIQ32X2NEHXCE6RVQQA6CTHTSFIKK3POKHAUW65WUA, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642301421 (505h44m22.513384466s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 17:13:22.642311620 (505h44m22.513394656s) [noise=true , address=45.91.116.4:4940 , piece_id=ZU3EXBGT6JSMNXFMFUE6475JKTOQTMKYHWPHEUCLJ2WTYKCRQ63A, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642589900 (505h44m22.513672955s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: context canceled; context canceled Dec 26 17:13:22.642599058 (505h44m22.513682093s) [noise=true , address=45.91.116.6:4976 , piece_id=HOPVSAQ73M56O2I227FTTS7C7M7L2ULDCW4PDCJ433F3S2ED37RQ, seg_pos={0 0}, upload=443784]: Long tail cancellation due to optimal threshold Dec 26 17:13:22.642620228 (505h44m22.513703273s) [seg_pos={0 0}, upload=443784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:13:22.642698367 (505h44m22.513781412s) [upload=443784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:13:22.642705129 (505h44m22.513788175s) [upload=443784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:13:22.932721441 (505h44m22.803804476s) [upload=443784]: Done uploading segments... Dec 26 17:13:22.932740437 (505h44m22.803823482s) [upload=443784]: Upload finished. err: <nil> Duration: 30.001954693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......X....X...............................] duration: 50.283932746s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:13:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 50.283949257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X......XXX...X........X..................................] duration: 3.369785414s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:13:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.369812375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...............X.......X....X..............................] duration: 54.486857909s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:12:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 54.486886302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X......XXX...X........X.................................] duration: 3.539637442s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:12:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539656789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX..] duration: 6.44131592s trace: https://jaeger.dev.storj.io/trace/2a33c8a0896e2812 FAIL (2025-12-26 17:12:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:09:59.412405166 (505h40m59.283488221s) [upload=443734]: Starting upload Dec 26 17:09:59.412488264 (505h40m59.283571299s) [upload=443734]: Done starting upload Dec 26 17:09:59.412584588 (505h40m59.283667633s) [upload=443734]: Uploading segments... Dec 26 17:09:59.412702623 (505h40m59.283785668s) [upload=443734]: Got next segment. Inline: false Dec 26 17:09:59.412739273 (505h40m59.283822318s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:09:59.412746376 (505h40m59.283829431s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:09:59.656808823 (505h40m59.527891868s) [seg_pos={0 0}, upload=443734]: Begin upload segment... Dec 26 17:09:59.656851325 (505h40m59.52793436s) [seg_pos={0 0}, upload=443734]: Done begin upload segment. Dec 26 17:09:59.656875851 (505h40m59.527958896s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.656927179 (505h40m59.528010224s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657442641 (505h40m59.528525676s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657819287 (505h40m59.528902322s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657989451 (505h40m59.529072486s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658360388 (505h40m59.529443433s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658632737 (505h40m59.529715772s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658977594 (505h40m59.530060639s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.661716348 (505h40m59.532799393s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.662944547 (505h40m59.534027592s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.663187159 (505h40m59.534270194s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664207722 (505h40m59.535290767s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664395620 (505h40m59.535478665s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.665648896 (505h40m59.536731931s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.667001091 (505h40m59.538084126s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.668250921 (505h40m59.539333956s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:10:01.276948018 (505h41m1.148031064s) [upload=443734]: Next returned nil segment Dec 26 17:10:01.276989468 (505h41m1.148072513s) [upload=443734]: Waiting for error group managing segments... Dec 26 17:10:01.660212240 (505h41m1.531295285s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.667666294 (505h41m1.538749329s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.670284259 (505h41m1.541367304s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.680257970 (505h41m1.551341016s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.683131332 (505h41m1.554214377s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.687154904 (505h41m1.558237969s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.696419345 (505h41m1.56750239s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.724453487 (505h41m1.595536532s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.916593334 (505h41m1.787676399s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.917178638 (505h41m1.788261683s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.918198120 (505h41m1.789281165s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.050222307 (505h41m1.921305362s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055406538 (505h41m1.926489583s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055439792 (505h41m1.926522837s) [seg_pos={0 0}, upload=443734]: Segment reached optimal threshold of 13 pieces. Dec 26 17:10:02.055713814 (505h41m1.926796869s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.055726138 (505h41m1.926809213s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056135126 (505h41m1.927218181s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056147029 (505h41m1.927230074s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056404779 (505h41m1.927487835s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056416652 (505h41m1.927499727s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056520950 (505h41m1.927604035s) [seg_pos={0 0}, upload=443734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:10:02.056636571 (505h41m1.927719616s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:10:02.056644276 (505h41m1.927727321s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:10:02.409013006 (505h41m2.280096051s) [upload=443734]: Done uploading segments... Dec 26 17:10:02.409034688 (505h41m2.280117733s) [upload=443734]: Upload finished. err: <nil> Duration: 6.441345376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...............X.......X....X.............................] duration: 42.531558931s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:11:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 42.531572937s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X......XXX...X........X................................] duration: 3.526666924s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:11:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526694897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX.] duration: 9.919684705s trace: https://jaeger.dev.storj.io/trace/2a33c8a0896e2812 FAIL (2025-12-26 17:11:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:09:59.412405166 (505h40m59.283488221s) [upload=443734]: Starting upload Dec 26 17:09:59.412488264 (505h40m59.283571299s) [upload=443734]: Done starting upload Dec 26 17:09:59.412584588 (505h40m59.283667633s) [upload=443734]: Uploading segments... Dec 26 17:09:59.412702623 (505h40m59.283785668s) [upload=443734]: Got next segment. Inline: false Dec 26 17:09:59.412739273 (505h40m59.283822318s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:09:59.412746376 (505h40m59.283829431s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:09:59.656808823 (505h40m59.527891868s) [seg_pos={0 0}, upload=443734]: Begin upload segment... Dec 26 17:09:59.656851325 (505h40m59.52793436s) [seg_pos={0 0}, upload=443734]: Done begin upload segment. Dec 26 17:09:59.656875851 (505h40m59.527958896s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.656927179 (505h40m59.528010224s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657442641 (505h40m59.528525676s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657819287 (505h40m59.528902322s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657989451 (505h40m59.529072486s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658360388 (505h40m59.529443433s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658632737 (505h40m59.529715772s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658977594 (505h40m59.530060639s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.661716348 (505h40m59.532799393s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.662944547 (505h40m59.534027592s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.663187159 (505h40m59.534270194s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664207722 (505h40m59.535290767s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664395620 (505h40m59.535478665s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.665648896 (505h40m59.536731931s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.667001091 (505h40m59.538084126s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.668250921 (505h40m59.539333956s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:10:01.276948018 (505h41m1.148031064s) [upload=443734]: Next returned nil segment Dec 26 17:10:01.276989468 (505h41m1.148072513s) [upload=443734]: Waiting for error group managing segments... Dec 26 17:10:01.660212240 (505h41m1.531295285s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.667666294 (505h41m1.538749329s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.670284259 (505h41m1.541367304s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.680257970 (505h41m1.551341016s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.683131332 (505h41m1.554214377s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.687154904 (505h41m1.558237969s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.696419345 (505h41m1.56750239s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.724453487 (505h41m1.595536532s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.916593334 (505h41m1.787676399s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.917178638 (505h41m1.788261683s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.918198120 (505h41m1.789281165s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.050222307 (505h41m1.921305362s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055406538 (505h41m1.926489583s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055439792 (505h41m1.926522837s) [seg_pos={0 0}, upload=443734]: Segment reached optimal threshold of 13 pieces. Dec 26 17:10:02.055713814 (505h41m1.926796869s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.055726138 (505h41m1.926809213s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056135126 (505h41m1.927218181s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056147029 (505h41m1.927230074s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056404779 (505h41m1.927487835s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056416652 (505h41m1.927499727s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056520950 (505h41m1.927604035s) [seg_pos={0 0}, upload=443734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:10:02.056636571 (505h41m1.927719616s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:10:02.056644276 (505h41m1.927727321s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:10:02.409013006 (505h41m2.280096051s) [upload=443734]: Done uploading segments... Dec 26 17:10:02.409034688 (505h41m2.280117733s) [upload=443734]: Upload finished. err: <nil> Duration: 9.919721945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...............X.......X....X............................] duration: 25.26759861s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:10:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 25.267627345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......XX] duration: 30.004006056s trace: https://jaeger.dev.storj.io/trace/2a33c8a0896e2812 FAIL (2025-12-26 17:10:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 17:09:59.412405166 (505h40m59.283488221s) [upload=443734]: Starting upload Dec 26 17:09:59.412488264 (505h40m59.283571299s) [upload=443734]: Done starting upload Dec 26 17:09:59.412584588 (505h40m59.283667633s) [upload=443734]: Uploading segments... Dec 26 17:09:59.412702623 (505h40m59.283785668s) [upload=443734]: Got next segment. Inline: false Dec 26 17:09:59.412739273 (505h40m59.283822318s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:09:59.412746376 (505h40m59.283829431s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:09:59.656808823 (505h40m59.527891868s) [seg_pos={0 0}, upload=443734]: Begin upload segment... Dec 26 17:09:59.656851325 (505h40m59.52793436s) [seg_pos={0 0}, upload=443734]: Done begin upload segment. Dec 26 17:09:59.656875851 (505h40m59.527958896s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.656927179 (505h40m59.528010224s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657442641 (505h40m59.528525676s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657819287 (505h40m59.528902322s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.657989451 (505h40m59.529072486s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658360388 (505h40m59.529443433s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658632737 (505h40m59.529715772s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.658977594 (505h40m59.530060639s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.661716348 (505h40m59.532799393s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.662944547 (505h40m59.534027592s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.663187159 (505h40m59.534270194s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664207722 (505h40m59.535290767s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.664395620 (505h40m59.535478665s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.665648896 (505h40m59.536731931s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.667001091 (505h40m59.538084126s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:09:59.668250921 (505h40m59.539333956s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Uploading piece... Dec 26 17:10:01.276948018 (505h41m1.148031064s) [upload=443734]: Next returned nil segment Dec 26 17:10:01.276989468 (505h41m1.148072513s) [upload=443734]: Waiting for error group managing segments... Dec 26 17:10:01.660212240 (505h41m1.531295285s) [noise=true , address=145.79.227.48:4301 , piece_id=UBLR4FPGYXW3QGW5RINDRQU34DFMJ62KXTF7USITCS5AEMVZWURA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.667666294 (505h41m1.538749329s) [noise=true , address=145.79.227.38:4978 , piece_id=UGJRYNXKGL7ARWFG45JFYNHHP362R4HGH4JRN4DE72ZI7RLQ72EQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.670284259 (505h41m1.541367304s) [noise=true , address=145.79.227.8:4330 , piece_id=2X6YFSNCN64EFHJKY4VVREPZJMAPY6RSHM7CWBGHMAXKT6CKUEEA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.680257970 (505h41m1.551341016s) [noise=true , address=145.79.227.50:4375 , piece_id=OZQWOY5C4XIBDFACEJQLMONB4O4NZ6GSEWF44B73HFUZHAJ5JMAA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.683131332 (505h41m1.554214377s) [noise=true , address=145.79.227.36:4517 , piece_id=P4WMX6REVK3NSEHTRASQH2243DDSNRNTB7RZFKFCRIRJPY64A6GQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.687154904 (505h41m1.558237969s) [noise=true , address=145.79.227.13:4804 , piece_id=HGHJAR7O3ZMOA7I6O46LH37P4ONDHFEZKALDRPAL4JRLQRWRDW6A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.696419345 (505h41m1.56750239s) [noise=true , address=145.79.227.51:4411 , piece_id=B6CBEBYDQFVLT5UVBUFTWXCHSIKSSUUZMIQPMERSU42T7GW2H5PQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.724453487 (505h41m1.595536532s) [noise=true , address=145.79.227.12:4765 , piece_id=LJKCGEW5WNIMFCHB46QO25RCNYA5N4FXNXBBHU7GI5ISICJQCMLA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.916593334 (505h41m1.787676399s) [noise=true , address=77.90.31.21:4878 , piece_id=4C7OQEEYO2QRF6KSUGXL2FE7454H3NQUFU5RFF2HJ7SPOWWF4YZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.917178638 (505h41m1.788261683s) [noise=true , address=45.91.116.5:4299 , piece_id=RZG6KTP2LHGXI5P2XG6EAQWC3LFH656J3WUN6P56SDE7GL46VU3A, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:01.918198120 (505h41m1.789281165s) [noise=true , address=77.90.31.10:4519 , piece_id=YSVQZKL5KLG7B4X5EWYGW4TLI46JKZCPYA5D35C2CWJ5FQSCKKFA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.050222307 (505h41m1.921305362s) [noise=true , address=145.79.227.2:4113 , piece_id=FZGPJMS42XXXHMLCMLTVKQRREG3FRAMLHYLEREC2ACGOETGOLYXQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055406538 (505h41m1.926489583s) [noise=true , address=45.91.116.4:4939 , piece_id=7MV6TYBD2Y5KXYG2JH2L4HOPZVTDL5HT6S3AXVELUPW3QR3BI5ZA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: <nil> Dec 26 17:10:02.055439792 (505h41m1.926522837s) [seg_pos={0 0}, upload=443734]: Segment reached optimal threshold of 13 pieces. Dec 26 17:10:02.055713814 (505h41m1.926796869s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.055726138 (505h41m1.926809213s) [noise=true , address=45.91.116.10:4084 , piece_id=JBTFBXZD64NAY7IZ5VMY4RRNMMNGE7HYMFHJMBPXJRTWQIRQLZ4Q, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056135126 (505h41m1.927218181s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056147029 (505h41m1.927230074s) [noise=true , address=45.91.116.12:4226 , piece_id=DKRH6VWXKKT33JJO7BHTS7PZBRTH63YA5DPEIWSOXSWNZW2SFDHA, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056404779 (505h41m1.927487835s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 17:10:02.056416652 (505h41m1.927499727s) [noise=true , address=77.90.31.17:4770 , piece_id=7Z4ROAPSSWS3VMVSME47UIYO7NOPEOWDH3XSDSXZRMUBVXS2XVOQ, seg_pos={0 0}, upload=443734]: Long tail cancellation due to optimal threshold Dec 26 17:10:02.056520950 (505h41m1.927604035s) [seg_pos={0 0}, upload=443734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:10:02.056636571 (505h41m1.927719616s) [upload=443734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:10:02.056644276 (505h41m1.927727321s) [upload=443734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:10:02.409013006 (505h41m2.280096051s) [upload=443734]: Done uploading segments... Dec 26 17:10:02.409034688 (505h41m2.280117733s) [upload=443734]: Upload finished. err: <nil> Duration: 30.004017388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X...............X.......X....X...........................] duration: 42.368620661s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:10:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 42.368651541s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X......XXX...X........X...............................] duration: 3.890908523s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:10:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.890968307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......X] duration: 30.003075515s trace: https://jaeger.dev.storj.io/trace/7fa74dcd6777d738 FAIL (2025-12-26 17:09:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 2V35BJ3SDKMY3GWKTC6MGMPDPKSXQGUYV4OQ2SE747T6KCMSCIWA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: MNHP3ICGPDFXB5CBBVWQHF365GOQPST46JDUBI6BT2UGWRFK6WOQ) context deadline exceeded ======LOGS====== Dec 26 17:09:29.133151177 (505h40m29.004234222s) [upload=443729]: Starting upload Dec 26 17:09:29.133227001 (505h40m29.004310056s) [upload=443729]: Done starting upload Dec 26 17:09:29.133312353 (505h40m29.004395388s) [upload=443729]: Uploading segments... Dec 26 17:09:29.133467960 (505h40m29.004551005s) [upload=443729]: Got next segment. Inline: false Dec 26 17:09:29.133505902 (505h40m29.004588937s) [upload=443729]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:09:29.133511061 (505h40m29.004594117s) [upload=443729]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:09:29.385162329 (505h40m29.256245364s) [seg_pos={0 0}, upload=443729]: Begin upload segment... Dec 26 17:09:29.385217574 (505h40m29.256300609s) [seg_pos={0 0}, upload=443729]: Done begin upload segment. Dec 26 17:09:29.385297316 (505h40m29.256380361s) [noise=true , address=145.79.227.49:4338 , piece_id=R5JMKO5GTTPVZEU2NF2KCTQHVSRBJHLEA4XLO2WALKTWSHZ25TZQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.385311804 (505h40m29.256394839s) [noise=true , address=145.79.227.5:4222 , piece_id=SM5EUHIF3TUKSNTJFYIAPJX3GK56PHCPNM6BYNYFQIU5AXO6VOOA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386019221 (505h40m29.257102266s) [noise=true , address=77.90.31.15:4700 , piece_id=MNHP3ICGPDFXB5CBBVWQHF365GOQPST46JDUBI6BT2UGWRFK6WOQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386171460 (505h40m29.257254505s) [noise=true , address=145.79.227.52:4446 , piece_id=267V44INJDC35GFAVI3SF3ABEXZ2EEWCBKZNB2BMXXUZPA3CBUKA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386581772 (505h40m29.257664807s) [noise=true , address=145.79.227.42:4122 , piece_id=ND6SE7IYN2I7MEXE3XWY4TLHVR62OV6YMUD7RFTM4QWME4FVJYNQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386619003 (505h40m29.257702038s) [noise=true , address=145.79.227.38:4977 , piece_id=5HZR7ZNURSLY2WIRTL4V7QODZT4RCAX2A2EW6QG5OEJ72AKGJOHQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.386876864 (505h40m29.257959899s) [noise=true , address=145.79.227.3:4149 , piece_id=UK5YDQW3ISQ4DGUVICJXSHWQYOMVD2XTGFULMAD5HGZS7M36TIHA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.387152179 (505h40m29.258235214s) [noise=true , address=45.91.116.6:4974 , piece_id=2V35BJ3SDKMY3GWKTC6MGMPDPKSXQGUYV4OQ2SE747T6KCMSCIWA, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.387508908 (505h40m29.258591953s) [noise=true , address=145.79.227.47:4266 , piece_id=CV7AYDKW2QU36URGSMOUIM6JO4EU7AJA7HB2WQWXQ3Q4NYAX3WKQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.387793720 (505h40m29.258876755s) [noise=true , address=145.79.227.46:4601 , piece_id=GGQ7LNBL2C2YUD42SBTTNDKTTAYTJI2VMD3E35F2HNNSPVOPC46A, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.388066008 (505h40m29.259149054s) [noise=true , address=45.91.116.8:4047 , piece_id=VIW572WRWPYFXDDQIMODBEC676X6TRKHL4X6TK332MMS3X5E2Q4A, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.388185926 (505h40m29.259268971s) [noise=true , address=145.79.227.50:4374 , piece_id=P6WZAIBQQWA535GX4SGGPCJC6HDZYHMM4JBOXWFWPYZRYRLXJHLQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.392835450 (505h40m29.263918485s) [noise=true , address=77.90.31.14:4663 , piece_id=GVKAGH6B37ZXF5C5ITQZO726KRGDWTD4W3KFCLYQD6JKKNROOIEQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.393323039 (505h40m29.264406074s) [noise=true , address=45.91.116.12:4228 , piece_id=BDKXMPTAJSLEMKHRVX5QNRE6H57URA3JSJILN2RWQ5GFFKAD6GAQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.394641019 (505h40m29.265724064s) [noise=true , address=145.79.227.12:4767 , piece_id=SPVO53RPSN3JFWIG4HYD2BB4XIZ2F7AAGJYA7EYYJWDDN22LDOIQ, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:29.395598301 (505h40m29.266681346s) [noise=true , address=45.91.116.14:4190 , piece_id=LA7AEL6DNGQLBQHVTF7SGMQLVZNF5BOYY7EOZIPIC5YSNIUHXQ3Q, seg_pos={0 0}, upload=443729]: Uploading piece... Dec 26 17:09:31.005665530 (505h40m30.876748575s) [upload=443729]: Next returned nil segment Dec 26 17:09:31.005677944 (505h40m30.876760989s) [upload=443729]: Waiting for error group managing segments... Dec 26 17:09:31.389076481 (505h40m31.260159516s) [noise=true , address=145.79.227.52:4446 , piece_id=267V44INJDC35GFAVI3SF3ABEXZ2EEWCBKZNB2BMXXUZPA3CBUKA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.391690869 (505h40m31.262773904s) [noise=true , address=145.79.227.46:4601 , piece_id=GGQ7LNBL2C2YUD42SBTTNDKTTAYTJI2VMD3E35F2HNNSPVOPC46A, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.396665723 (505h40m31.267748768s) [noise=true , address=145.79.227.5:4222 , piece_id=SM5EUHIF3TUKSNTJFYIAPJX3GK56PHCPNM6BYNYFQIU5AXO6VOOA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.398209442 (505h40m31.269292487s) [noise=true , address=145.79.227.49:4338 , piece_id=R5JMKO5GTTPVZEU2NF2KCTQHVSRBJHLEA4XLO2WALKTWSHZ25TZQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.400697820 (505h40m31.271780866s) [noise=true , address=145.79.227.38:4977 , piece_id=5HZR7ZNURSLY2WIRTL4V7QODZT4RCAX2A2EW6QG5OEJ72AKGJOHQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.408174808 (505h40m31.279257853s) [noise=true , address=145.79.227.50:4374 , piece_id=P6WZAIBQQWA535GX4SGGPCJC6HDZYHMM4JBOXWFWPYZRYRLXJHLQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.409061566 (505h40m31.280144611s) [noise=true , address=145.79.227.42:4122 , piece_id=ND6SE7IYN2I7MEXE3XWY4TLHVR62OV6YMUD7RFTM4QWME4FVJYNQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.417519532 (505h40m31.288602567s) [noise=true , address=145.79.227.3:4149 , piece_id=UK5YDQW3ISQ4DGUVICJXSHWQYOMVD2XTGFULMAD5HGZS7M36TIHA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.424664949 (505h40m31.295747984s) [noise=true , address=145.79.227.47:4266 , piece_id=CV7AYDKW2QU36URGSMOUIM6JO4EU7AJA7HB2WQWXQ3Q4NYAX3WKQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.425492514 (505h40m31.296575549s) [noise=true , address=145.79.227.12:4767 , piece_id=SPVO53RPSN3JFWIG4HYD2BB4XIZ2F7AAGJYA7EYYJWDDN22LDOIQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.664797552 (505h40m31.535880597s) [noise=true , address=45.91.116.6:4974 , piece_id=2V35BJ3SDKMY3GWKTC6MGMPDPKSXQGUYV4OQ2SE747T6KCMSCIWA, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.682185633 (505h40m31.553268668s) [noise=true , address=77.90.31.14:4663 , piece_id=GVKAGH6B37ZXF5C5ITQZO726KRGDWTD4W3KFCLYQD6JKKNROOIEQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.706708498 (505h40m31.577791543s) [noise=true , address=77.90.31.15:4700 , piece_id=MNHP3ICGPDFXB5CBBVWQHF365GOQPST46JDUBI6BT2UGWRFK6WOQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: <nil> Dec 26 17:09:31.706727965 (505h40m31.577811011s) [seg_pos={0 0}, upload=443729]: Segment reached optimal threshold of 13 pieces. Dec 26 17:09:31.706934148 (505h40m31.578017193s) [noise=true , address=45.91.116.8:4047 , piece_id=VIW572WRWPYFXDDQIMODBEC676X6TRKHL4X6TK332MMS3X5E2Q4A, seg_pos={0 0}, upload=443729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 17:09:31.706941081 (505h40m31.578024116s) [noise=true , address=45.91.116.8:4047 , piece_id=VIW572WRWPYFXDDQIMODBEC676X6TRKHL4X6TK332MMS3X5E2Q4A, seg_pos={0 0}, upload=443729]: Long tail cancellation due to optimal threshold Dec 26 17:09:31.707030611 (505h40m31.578113646s) [noise=true , address=45.91.116.14:4190 , piece_id=LA7AEL6DNGQLBQHVTF7SGMQLVZNF5BOYY7EOZIPIC5YSNIUHXQ3Q, seg_pos={0 0}, upload=443729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 17:09:31.707038447 (505h40m31.578121482s) [noise=true , address=45.91.116.14:4190 , piece_id=LA7AEL6DNGQLBQHVTF7SGMQLVZNF5BOYY7EOZIPIC5YSNIUHXQ3Q, seg_pos={0 0}, upload=443729]: Long tail cancellation due to optimal threshold Dec 26 17:09:31.707187059 (505h40m31.578270105s) [noise=true , address=45.91.116.12:4228 , piece_id=BDKXMPTAJSLEMKHRVX5QNRE6H57URA3JSJILN2RWQ5GFFKAD6GAQ, seg_pos={0 0}, upload=443729]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 17:09:31.707193411 (505h40m31.578276446s) [noise=true , address=45.91.116.12:4228 , piece_id=BDKXMPTAJSLEMKHRVX5QNRE6H57URA3JSJILN2RWQ5GFFKAD6GAQ, seg_pos={0 0}, upload=443729]: Long tail cancellation due to optimal threshold Dec 26 17:09:31.707217648 (505h40m31.578300683s) [seg_pos={0 0}, upload=443729]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:09:31.707296969 (505h40m31.578380004s) [upload=443729]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:09:31.707301958 (505h40m31.578384993s) [upload=443729]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:09:32.015660245 (505h40m31.88674329s) [upload=443729]: Done uploading segments... Dec 26 17:09:32.015718627 (505h40m31.886801662s) [upload=443729]: Upload finished. err: <nil> Duration: 30.003086706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X......] duration: 28.674908803s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:09:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 28.674937548s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X......XXX...X........X..............................] duration: 3.430116345s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:09:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430143326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X...............X.......X....X..........................] duration: 30.634200625s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:08:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 30.63425041s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........X......XXX...X........X.............................] duration: 3.452427401s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:08:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452454311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X.....] duration: 7.473619703s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:08:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 7.473647777s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........X......XXX...X........X............................] duration: 3.490434331s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:08:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490467945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 7.724234689s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:07:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.724258975s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X......XXX...X........X...........................] duration: 3.545993906s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:07:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546018764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 21.472964818s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:07:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 21.472995765s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X...............X.......X....X.........................] duration: 30.321610847s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:07:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 30.321640295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X....] duration: 6.126098129s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 6.126131202s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.139869508s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:07:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.139898913s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..X...............X.......X....X........................] duration: 29.326064837s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:07:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 29.326100797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X..........X......XXX...X........X..........................] duration: 3.671401743s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:06:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.671448954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X...] duration: 6.172151534s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:06:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 6.172179206s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XX..........X......XXX...X........X.........................] duration: 3.914215043s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:06:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.91434957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 8.704370947s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:06:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 8.704395354s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..X...............X.......X....X.......................] duration: 23.091033912s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:06:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 23.091054021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.XX..........X......XXX...X........X........................] duration: 3.469748642s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:05:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469770463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X...............X.......X....X......................] duration: 48.536851827s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:05:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 48.536879771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 9.497002584s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:05:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 9.497018644s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X..] duration: 5.954588957s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:05:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 5.954608194s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.693761601s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:05:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.693792992s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X.] duration: 25.868765259s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:05:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 25.868782752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..X...............X.......X....X.....................] duration: 35.523815844s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:04:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 35.523841952s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..XX..........X......XXX...X........X.......................] duration: 4.297019848s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:04:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.297046399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.X] duration: 30.002701801s trace: https://jaeger.dev.storj.io/trace/6913281d5a8e3dd0 FAIL (2025-12-26 17:04:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA) context deadline exceeded ======LOGS====== Dec 26 17:03:55.984035493 (505h34m55.855118538s) [upload=443642]: Starting upload Dec 26 17:03:55.984111207 (505h34m55.855194252s) [upload=443642]: Done starting upload Dec 26 17:03:55.984170070 (505h34m55.855253115s) [upload=443642]: Uploading segments... Dec 26 17:03:55.984292392 (505h34m55.855375437s) [upload=443642]: Got next segment. Inline: false Dec 26 17:03:55.984326667 (505h34m55.855409712s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:03:55.984331047 (505h34m55.855414082s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:03:56.238902253 (505h34m56.109985288s) [seg_pos={0 0}, upload=443642]: Begin upload segment... Dec 26 17:03:56.238949624 (505h34m56.110032659s) [seg_pos={0 0}, upload=443642]: Done begin upload segment. Dec 26 17:03:56.238975653 (505h34m56.110058698s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.239975767 (505h34m56.111058802s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.240805628 (505h34m56.111888663s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241443091 (505h34m56.112526126s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.241884562 (505h34m56.112967597s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242316605 (505h34m56.11339964s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.242691238 (505h34m56.113774283s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243020225 (505h34m56.11410326s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243371895 (505h34m56.11445494s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.243724927 (505h34m56.114807972s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244055576 (505h34m56.115138621s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244368251 (505h34m56.115451286s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.244686918 (505h34m56.115769963s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245023469 (505h34m56.116106504s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245331646 (505h34m56.116414681s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:56.245683746 (505h34m56.116766791s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Uploading piece... Dec 26 17:03:57.914457961 (505h34m57.785540996s) [upload=443642]: Next returned nil segment Dec 26 17:03:57.914493058 (505h34m57.785576103s) [upload=443642]: Waiting for error group managing segments... Dec 26 17:03:58.244623122 (505h34m58.115706157s) [noise=true , address=145.79.227.51:4409 , piece_id=TD4RM2SZ2NAXAXLOIS5RV6DT6AZLBUWYKDLKNBIEKON5T33Q3W7A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252354966 (505h34m58.123438001s) [noise=true , address=145.79.227.50:4373 , piece_id=MQJYDKMK6DC4K44L47AI2LYHYJU4IPQIWMUTL6AHYNPK4UP2QZDA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.252852312 (505h34m58.123935347s) [noise=true , address=145.79.227.22:4509 , piece_id=HZZJKI2CJBFO5MWGN2X2BTZEZANJGH3CJL7LYESR7NNABAIDKQMA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257518728 (505h34m58.128601773s) [noise=true , address=145.79.227.15:4874 , piece_id=MQNIOQ2PHDJLJHOZ5L5PU4NMU5HJXIN336JZ3BRBQ7NSMEXGDGMQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.257917567 (505h34m58.129000602s) [noise=true , address=145.79.227.46:4603 , piece_id=P2QBJUCSL3USGNB73NXPTZJ5IAYVSUMSVCM36XU3NIGGXWNDEIHA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.263624444 (505h34m58.134707489s) [noise=true , address=145.79.227.52:4446 , piece_id=ICT4QMIW45MPB6P6XGRSHUUKJGO73GH5DQHRLOOJRLP2CZ25AFFQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.264638234 (505h34m58.135721279s) [noise=true , address=145.79.227.36:4519 , piece_id=E4SROAJIWVTPCUC32FVUXCJPA63OESHYQGC7STTYCZLW4EOK365Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.268370161 (505h34m58.139453196s) [noise=true , address=145.79.227.11:4730 , piece_id=4FTT2FR55JDR7IRH23WHME5COVSAUB7X2MUUMBZSU4XYHYKDEQ5Q, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.271928316 (505h34m58.143011351s) [noise=true , address=145.79.227.41:4085 , piece_id=ARRMEESFUNTZH56W5PPYDT46JRYZ2YTU2B553LJC6B5MHSXQFOBQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.272206616 (505h34m58.143289661s) [noise=true , address=145.79.227.47:4267 , piece_id=OLAM26HPGCJDYA37WHLVW5TH354PE2WTVNANVCH44N5RZTWOKT5A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.558570102 (505h34m58.429653137s) [noise=true , address=77.90.31.17:4770 , piece_id=IUOEL7AFTUXPINRFBWU3AIZZVTE5P7XAJARFK57BTUL3PAIGMSJA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.564248515 (505h34m58.43533155s) [noise=true , address=77.90.31.19:4844 , piece_id=AJM4PV73XMOMOOHLA7SYQVMN767QBNLYQF2FZW7S3LWA5GC2NQRQ, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681634658 (505h34m58.552717704s) [noise=true , address=45.91.116.10:4083 , piece_id=TLIXCOJCTTAOETDFN4WOTY63P4J3JWKF6M2LL65TDUUKWGIFEVQA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: <nil> Dec 26 17:03:58.681654948 (505h34m58.552737983s) [seg_pos={0 0}, upload=443642]: Segment reached optimal threshold of 13 pieces. Dec 26 17:03:58.681888762 (505h34m58.552971797s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681895475 (505h34m58.55297851s) [noise=true , address=145.79.227.29:4941 , piece_id=IZAX564CVSTIKTFNMMPCDELGG2SXOYKGVGDG6YDTKL5IA7RRHDIA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.681923438 (505h34m58.553006473s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 17:03:58.681933819 (505h34m58.553016864s) [noise=true , address=145.79.227.33:4871 , piece_id=CTFAR33P3QBGTDXPLETLJT6DQS4S2YDHAOVR5LZ3TJ7Z7IQGPXRA, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682161562 (505h34m58.553244607s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:43792->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 17:03:58.682168074 (505h34m58.553251109s) [noise=true , address=45.91.116.5:4298 , piece_id=5R6N6KIYPOCDERF64LWHKWBVG2LCENAKHISMBZY3KXNYQZPUP33A, seg_pos={0 0}, upload=443642]: Long tail cancellation due to optimal threshold Dec 26 17:03:58.682222799 (505h34m58.553305834s) [seg_pos={0 0}, upload=443642]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:03:58.682303261 (505h34m58.553386296s) [upload=443642]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:03:58.682307930 (505h34m58.553390965s) [upload=443642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:03:58.990304800 (505h34m58.861387835s) [upload=443642]: Done uploading segments... Dec 26 17:03:58.990326381 (505h34m58.861409416s) [upload=443642]: Upload finished. err: <nil> Duration: 30.002735284s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...XX..........X......XXX...X........X......................] duration: 3.462098813s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:04:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462119362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 7.162924166s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 17:04:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.162955346s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X...............X.......X....X....................] duration: 38.902821399s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 17:03:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 38.902840134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....XX..........X......XXX...X........X.....................] duration: 3.531058065s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:03:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531073324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.X....XX...X...XXX..X..X...X...XXXX.......XX..X.X..XX..X.] duration: 16.409813192s trace: https://jaeger.dev.storj.io/trace/47a12fef265fe8e2 FAIL (2025-12-26 17:03:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: ANLKYPF5DXUQX3V7TOCJR2OSYZFN5XI7UAEBL37YKS346XOPHXDQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: JW7ENELF4G6AD5FIZKLB3XQGCJVLM5DV7CWCS5QXOHF46Z35YESA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7WQX56GLWRG6PZFP3DNF5COF6RYT52PHL3NTQOE5MJFL64BH62XQ) context deadline exceeded ======LOGS====== s... Dec 26 17:02:56.372426380 (505h33m56.243509415s) [upload=443629]: Got next segment. Inline: false Dec 26 17:02:56.372539757 (505h33m56.243622802s) [upload=443629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 17:02:56.372546510 (505h33m56.243629545s) [upload=443629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 17:02:56.618142927 (505h33m56.489225962s) [seg_pos={0 0}, upload=443629]: Begin upload segment... Dec 26 17:02:56.618218431 (505h33m56.489301476s) [seg_pos={0 0}, upload=443629]: Done begin upload segment. Dec 26 17:02:56.618281171 (505h33m56.489364206s) [noise=true , address=145.79.227.16:4912 , piece_id=5RO23B7ROXA7GZOKRQRARCNBRPKRZOQPD46B6CEDCW4DX5JUJJ4Q, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.619477068 (505h33m56.490560113s) [noise=true , address=145.79.227.29:4943 , piece_id=E35LVN5VXI4ALAYZCRHTURLMVC6YDAA6HUGRPA666CYKP4X3GPRA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.620043326 (505h33m56.491126382s) [noise=true , address=77.90.31.13:4629 , piece_id=ANLKYPF5DXUQX3V7TOCJR2OSYZFN5XI7UAEBL37YKS346XOPHXDQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.620486910 (505h33m56.491569955s) [noise=true , address=145.79.227.47:4265 , piece_id=HQOUAJFSOSD5ZZ7ZXMHWHIP5ATQFZXB5PJIJT6YLPJAH6AVHCMEA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.620834462 (505h33m56.491917497s) [noise=true , address=45.91.116.8:4045 , piece_id=JW7ENELF4G6AD5FIZKLB3XQGCJVLM5DV7CWCS5QXOHF46Z35YESA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.621178037 (505h33m56.492261082s) [noise=true , address=145.79.227.50:4374 , piece_id=2M2PILEN5KZAOFYXTTUI45FUOJIHACC6KOG2Q6REHKDTL6GODPJQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.621557860 (505h33m56.492640905s) [noise=true , address=45.91.116.13:4156 , piece_id=UTKKKGDUBOSBVTNX7HW2COHTTQABO7AOUDZNRBT4TWKRJWEL2MZA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.622087077 (505h33m56.493170123s) [noise=true , address=145.79.227.46:4603 , piece_id=XVI7WCCU5AG7IIRTY5M3UOERXRLUOYAQY3U2A7DQIPAOIDUX4VTA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.622512839 (505h33m56.493595884s) [noise=true , address=145.79.227.48:4301 , piece_id=MFE2O3E3RSHH7NMHPQB3MLPW4XA4NNNPYW3KABDHAFATHV4QNVTA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.622864158 (505h33m56.493947203s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVNHVUYNWHHBR2J4LTOHN4MOVH2DDX4DKWBPIHEWLZ5YHMJWNNYQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.623228782 (505h33m56.494311827s) [noise=true , address=145.79.227.45:4230 , piece_id=Z3AFJ67RIY4D7O5L5C7G26PQ7G3LHGAJJR36PEHT53TWSTGTSQBQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.623620557 (505h33m56.494703602s) [noise=true , address=77.90.31.10:4517 , piece_id=ALL5WRUOC7QSI2ZZ44JBMM5BICSKFRF5KDGTFXS6EKPOGJSN45SA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.624010571 (505h33m56.495093616s) [noise=true , address=45.91.116.5:4299 , piece_id=7WQX56GLWRG6PZFP3DNF5COF6RYT52PHL3NTQOE5MJFL64BH62XQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.624399321 (505h33m56.495482366s) [noise=true , address=145.79.227.22:4509 , piece_id=JX27J4R3XKYKR3LDIETJJ5USAWREA3CBZ5SDERD2PMLDJVDBDLYA, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.624816255 (505h33m56.49589929s) [noise=true , address=145.79.227.2:4114 , piece_id=RCKBOYQDY5ROW7ODK7ADVGWPT2MM753Z37V2SDDZDYG32NKC4FVQ, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:56.625131285 (505h33m56.49621432s) [noise=true , address=145.79.227.34:4905 , piece_id=KJRFUYCMHG3DGO33GAKGVK4KRSLSPR7WFAB5QLYJZTUIZZ6IMK7A, seg_pos={0 0}, upload=443629]: Uploading piece... Dec 26 17:02:58.327842451 (505h33m58.198925496s) [upload=443629]: Next returned nil segment Dec 26 17:02:58.327867940 (505h33m58.198950985s) [upload=443629]: Waiting for error group managing segments... Dec 26 17:02:58.628733781 (505h33m58.499816816s) [noise=true , address=145.79.227.45:4230 , piece_id=Z3AFJ67RIY4D7O5L5C7G26PQ7G3LHGAJJR36PEHT53TWSTGTSQBQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.634889603 (505h33m58.505972638s) [noise=true , address=145.79.227.22:4509 , piece_id=JX27J4R3XKYKR3LDIETJJ5USAWREA3CBZ5SDERD2PMLDJVDBDLYA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.636250083 (505h33m58.507333138s) [noise=true , address=145.79.227.50:4374 , piece_id=2M2PILEN5KZAOFYXTTUI45FUOJIHACC6KOG2Q6REHKDTL6GODPJQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.639680364 (505h33m58.51076341s) [noise=true , address=145.79.227.47:4265 , piece_id=HQOUAJFSOSD5ZZ7ZXMHWHIP5ATQFZXB5PJIJT6YLPJAH6AVHCMEA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.641408375 (505h33m58.51249142s) [noise=true , address=145.79.227.5:4222 , piece_id=ZVNHVUYNWHHBR2J4LTOHN4MOVH2DDX4DKWBPIHEWLZ5YHMJWNNYQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.644045357 (505h33m58.515128392s) [noise=true , address=145.79.227.46:4603 , piece_id=XVI7WCCU5AG7IIRTY5M3UOERXRLUOYAQY3U2A7DQIPAOIDUX4VTA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.644273631 (505h33m58.515356676s) [noise=true , address=145.79.227.2:4114 , piece_id=RCKBOYQDY5ROW7ODK7ADVGWPT2MM753Z37V2SDDZDYG32NKC4FVQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.649689564 (505h33m58.520772609s) [noise=true , address=145.79.227.48:4301 , piece_id=MFE2O3E3RSHH7NMHPQB3MLPW4XA4NNNPYW3KABDHAFATHV4QNVTA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.727898707 (505h33m58.598981742s) [noise=true , address=145.79.227.34:4905 , piece_id=KJRFUYCMHG3DGO33GAKGVK4KRSLSPR7WFAB5QLYJZTUIZZ6IMK7A, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:58.923049338 (505h33m58.794132383s) [noise=true , address=77.90.31.13:4629 , piece_id=ANLKYPF5DXUQX3V7TOCJR2OSYZFN5XI7UAEBL37YKS346XOPHXDQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.019884739 (505h33m58.890967784s) [noise=true , address=77.90.31.10:4517 , piece_id=ALL5WRUOC7QSI2ZZ44JBMM5BICSKFRF5KDGTFXS6EKPOGJSN45SA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.047118767 (505h33m58.918201802s) [noise=true , address=45.91.116.5:4299 , piece_id=7WQX56GLWRG6PZFP3DNF5COF6RYT52PHL3NTQOE5MJFL64BH62XQ, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.079095427 (505h33m58.950178462s) [noise=true , address=45.91.116.8:4045 , piece_id=JW7ENELF4G6AD5FIZKLB3XQGCJVLM5DV7CWCS5QXOHF46Z35YESA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: <nil> Dec 26 17:02:59.079114894 (505h33m58.950197929s) [seg_pos={0 0}, upload=443629]: Segment reached optimal threshold of 13 pieces. Dec 26 17:02:59.079288835 (505h33m58.95037188s) [noise=true , address=145.79.227.29:4943 , piece_id=E35LVN5VXI4ALAYZCRHTURLMVC6YDAA6HUGRPA666CYKP4X3GPRA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 26 17:02:59.079297371 (505h33m58.950380416s) [noise=true , address=145.79.227.29:4943 , piece_id=E35LVN5VXI4ALAYZCRHTURLMVC6YDAA6HUGRPA666CYKP4X3GPRA, seg_pos={0 0}, upload=443629]: Long tail cancellation due to optimal threshold Dec 26 17:02:59.079433621 (505h33m58.950516676s) [noise=true , address=145.79.227.16:4912 , piece_id=5RO23B7ROXA7GZOKRQRARCNBRPKRZOQPD46B6CEDCW4DX5JUJJ4Q, seg_pos={0 0}, upload=443629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:33400->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:33400->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33400->145.79.227.16:4912: use of closed network connection Dec 26 17:02:59.079449230 (505h33m58.950532265s) [noise=true , address=145.79.227.16:4912 , piece_id=5RO23B7ROXA7GZOKRQRARCNBRPKRZOQPD46B6CEDCW4DX5JUJJ4Q, seg_pos={0 0}, upload=443629]: Long tail cancellation due to optimal threshold Dec 26 17:02:59.079559660 (505h33m58.950642705s) [noise=true , address=45.91.116.13:4156 , piece_id=UTKKKGDUBOSBVTNX7HW2COHTTQABO7AOUDZNRBT4TWKRJWEL2MZA, seg_pos={0 0}, upload=443629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:45836->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:45836->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45836->45.91.116.13:4156: use of closed network connection Dec 26 17:02:59.079569590 (505h33m58.950652625s) [noise=true , address=45.91.116.13:4156 , piece_id=UTKKKGDUBOSBVTNX7HW2COHTTQABO7AOUDZNRBT4TWKRJWEL2MZA, seg_pos={0 0}, upload=443629]: Long tail cancellation due to optimal threshold Dec 26 17:02:59.079589898 (505h33m58.950672933s) [seg_pos={0 0}, upload=443629]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 17:02:59.079665502 (505h33m58.950748537s) [upload=443629]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:02:59.079671474 (505h33m58.950754509s) [upload=443629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:02:59.375961150 (505h33m59.247044195s) [upload=443629]: Done uploading segments... Dec 26 17:02:59.375986308 (505h33m59.247069343s) [upload=443629]: Upload finished. err: <nil> Duration: 16.409830254s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....XX..........X......XXX...X........X....................] duration: 3.502484356s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 17:03:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502511577s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)