Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXX] duration: 5.001775572s trace: https://jaeger.dev.storj.io/trace/3216a1fd591f991c FAIL (2025-12-28 12:49:11 UTC) orbiter warning: context deadline exceeded Duration: 5.001801381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X..XX.X....XX.X.....XX..............X.X.......X.X.....] duration: 6.436199333s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:48:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 6.436240381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 31.45242217s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:48:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 31.452450834s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [.XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXX] duration: 5.001888415s trace: https://jaeger.dev.storj.io/trace/6dbe38e012422341 FAIL (2025-12-28 12:48:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001917951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X..XX.X....XX.X.....XX..............X.X.......X.X....] duration: 6.111598631s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 6.11164061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 51.152614228s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:48:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 51.152643233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [..XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XX] duration: 5.001400516s trace: https://jaeger.dev.storj.io/trace/6431b5d80c1ae63a FAIL (2025-12-28 12:47:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001427438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 54.781741259s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:47:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 54.781771386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........X..XX.X....XX.X.....XX..............X.X.......X.X...] duration: 5.658014129s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:47:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 5.658042222s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....X] duration: 5.000838688s trace: https://jaeger.dev.storj.io/trace/57bd1b1a323dd497 FAIL (2025-12-28 12:47:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000865699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [X...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....] duration: 3.648202861s trace: https://jaeger.dev.storj.io/trace/58701863bd9aab9e FAIL (2025-12-28 12:46:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.648225513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........X..XX.X....XX.X.....XX..............X.X.......X.X..] duration: 5.923063387s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:46:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 5.923090509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 48.052269042s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:46:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 48.052285272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X...] duration: 3.494128126s trace: https://jaeger.dev.storj.io/trace/58701863bd9aab9e FAIL (2025-12-28 12:45:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494159405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.........X..XX.X....XX.X.....XX..............X.X.......X.X.] duration: 7.574991677s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:45:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 7.575008729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.........X..XX.X....XX.X.....XX..............X.X.......X.X] duration: 30.002655958s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:45:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 30.002778931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 38.283862378s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:45:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 38.283886333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X..] duration: 5.001936035s trace: https://jaeger.dev.storj.io/trace/58701863bd9aab9e FAIL (2025-12-28 12:45:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00196433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 48.13597118s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:44:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 48.13599734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.........X..XX.X....XX.X.....XX..............X.X.......X.] duration: 5.768241419s trace: https://jaeger.dev.storj.io/trace/b260c7acf5eda5 FAIL (2025-12-28 12:44:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: SHPKVZS3T5GPYXUOWO7AFZG2IMKLIVVF7VCFF2LTD7UE3PGJ2VLA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: ITLQDSDXS7SAEFCSDIR2LOHRDZ3TOTBOBXJ6MDIJWXZOFWKWLQ6Q) context deadline exceeded ======LOGS====== alse Dec 28 12:43:22.092912096 (549h14m21.963995131s) [upload=481570]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:43:22.092917186 (549h14m21.964000231s) [upload=481570]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:43:22.339171386 (549h14m22.210254441s) [seg_pos={0 0}, upload=481570]: Begin upload segment... Dec 28 12:43:22.339236040 (549h14m22.210319075s) [seg_pos={0 0}, upload=481570]: Done begin upload segment. Dec 28 12:43:22.339268501 (549h14m22.210351546s) [noise=true , address=145.79.227.37:4567 , piece_id=UIRTJ6KCOFKC4NQRH35SYPUGQOZKGGV3HRHKXAGXJUHJTR2TAGOA, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.339279061 (549h14m22.210362096s) [noise=true , address=145.79.227.36:4517 , piece_id=LNP7WULCMEXLLQQ5YC6C5HTYPOA33K2ZT6W5BCQTN26OQY3BIOIQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.339352160 (549h14m22.210435195s) [noise=true , address=145.79.227.39:4014 , piece_id=DIPPF7NGJQABWKAH3KQMA3UI34IODDVDZ47XOH5IPQUKI6CTY5UQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.340027235 (549h14m22.21111028s) [noise=true , address=77.90.31.19:4841 , piece_id=SHPKVZS3T5GPYXUOWO7AFZG2IMKLIVVF7VCFF2LTD7UE3PGJ2VLA, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.340041874 (549h14m22.211124929s) [noise=true , address=45.91.116.4:4939 , piece_id=7ALA7BX7S7BNWJWVBHYFFI4D3L2QUGIKJT3YFXGLI3EATXKWUVVQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.340618712 (549h14m22.211701757s) [noise=true , address=45.91.116.12:4227 , piece_id=DTKN5KTHQE4K2HDOL4EMMDNRVXWE2TJE774IYOYVZDDUWC76HESQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.340680129 (549h14m22.211763174s) [noise=true , address=45.91.116.10:4082 , piece_id=PLQEADE7EJM3XM7CJP4CZJ45Z7UQDXEXI4Z7NH32TGTAPLSJRHVQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.341129675 (549h14m22.21221271s) [noise=true , address=45.91.116.8:4045 , piece_id=DTHRCPL4INBRDDEULKI7ZL27QMRICKEDB5PLST4ARBLQ7AK4W57Q, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.341140666 (549h14m22.212223701s) [noise=true , address=145.79.227.42:4122 , piece_id=4RWNA3RY2IXYPLHGQ2TR43LKFSKIAYLCUI3MVIWU3KVTISYH4UVQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.341591614 (549h14m22.212674659s) [noise=true , address=77.90.31.21:4877 , piece_id=ITLQDSDXS7SAEFCSDIR2LOHRDZ3TOTBOBXJ6MDIJWXZOFWKWLQ6Q, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.341626571 (549h14m22.212709606s) [noise=true , address=45.91.116.11:4118 , piece_id=Q6DAYRJ5Q2XVV5OD44HLQUSR234EVNYU44DSET2LBJAVZKJQBEPQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.342223167 (549h14m22.213306212s) [noise=true , address=145.79.227.46:4603 , piece_id=LJPUYPWCLIGI5BJTXRNRUSOOOHZABB46ENFUYXUBC23E2ITQ6XIQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.342484775 (549h14m22.21356782s) [noise=true , address=145.79.227.20:4439 , piece_id=HMOXT7RVRMWI7GXHYS4N4GPRYVBNH3FIISOR2MEZY6IOWUATATUA, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.342652365 (549h14m22.21373541s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJOEUPZ2EPA7O3TFKB2IL26I7PL3ETA4SCFKG4KU77DTFLV674A, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.343146335 (549h14m22.21422937s) [noise=true , address=145.79.227.28:4726 , piece_id=LXWVUGMGSC4I4T5GN6ZRERJMCODK56KSBTIU32RTBD36RTZS4YVA, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.343616400 (549h14m22.214699455s) [noise=true , address=145.79.227.52:4445 , piece_id=BY5WCTBSZ7FGX73FN4UCMBTB2YQJJO2VYJC6K5QUAURUVXVPUBOQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:23.963232341 (549h14m23.834315386s) [upload=481570]: Next returned nil segment Dec 28 12:43:23.963249493 (549h14m23.834332528s) [upload=481570]: Waiting for error group managing segments... Dec 28 12:43:24.343816276 (549h14m24.214899341s) [noise=true , address=145.79.227.20:4439 , piece_id=HMOXT7RVRMWI7GXHYS4N4GPRYVBNH3FIISOR2MEZY6IOWUATATUA, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.346537077 (549h14m24.217620132s) [noise=true , address=145.79.227.39:4014 , piece_id=DIPPF7NGJQABWKAH3KQMA3UI34IODDVDZ47XOH5IPQUKI6CTY5UQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.348499945 (549h14m24.21958299s) [noise=true , address=145.79.227.46:4603 , piece_id=LJPUYPWCLIGI5BJTXRNRUSOOOHZABB46ENFUYXUBC23E2ITQ6XIQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.348628870 (549h14m24.219711915s) [noise=true , address=145.79.227.37:4567 , piece_id=UIRTJ6KCOFKC4NQRH35SYPUGQOZKGGV3HRHKXAGXJUHJTR2TAGOA, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.349147528 (549h14m24.220230573s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJOEUPZ2EPA7O3TFKB2IL26I7PL3ETA4SCFKG4KU77DTFLV674A, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.352771769 (549h14m24.223854804s) [noise=true , address=145.79.227.52:4445 , piece_id=BY5WCTBSZ7FGX73FN4UCMBTB2YQJJO2VYJC6K5QUAURUVXVPUBOQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.362662552 (549h14m24.233745598s) [noise=true , address=145.79.227.42:4122 , piece_id=4RWNA3RY2IXYPLHGQ2TR43LKFSKIAYLCUI3MVIWU3KVTISYH4UVQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.363249520 (549h14m24.234332555s) [noise=true , address=145.79.227.36:4517 , piece_id=LNP7WULCMEXLLQQ5YC6C5HTYPOA33K2ZT6W5BCQTN26OQY3BIOIQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.370520305 (549h14m24.24160334s) [noise=true , address=145.79.227.28:4726 , piece_id=LXWVUGMGSC4I4T5GN6ZRERJMCODK56KSBTIU32RTBD36RTZS4YVA, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.510838745 (549h14m24.38192179s) [noise=true , address=77.90.31.19:4841 , piece_id=SHPKVZS3T5GPYXUOWO7AFZG2IMKLIVVF7VCFF2LTD7UE3PGJ2VLA, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.653958389 (549h14m24.525041424s) [noise=true , address=77.90.31.21:4877 , piece_id=ITLQDSDXS7SAEFCSDIR2LOHRDZ3TOTBOBXJ6MDIJWXZOFWKWLQ6Q, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.689479327 (549h14m24.560562372s) [noise=true , address=45.91.116.8:4045 , piece_id=DTHRCPL4INBRDDEULKI7ZL27QMRICKEDB5PLST4ARBLQ7AK4W57Q, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.694874881 (549h14m24.565957926s) [noise=true , address=45.91.116.10:4082 , piece_id=PLQEADE7EJM3XM7CJP4CZJ45Z7UQDXEXI4Z7NH32TGTAPLSJRHVQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.694889939 (549h14m24.565972994s) [seg_pos={0 0}, upload=481570]: Segment reached optimal threshold of 13 pieces. Dec 28 12:43:24.695114016 (549h14m24.566197061s) [noise=true , address=45.91.116.4:4939 , piece_id=7ALA7BX7S7BNWJWVBHYFFI4D3L2QUGIKJT3YFXGLI3EATXKWUVVQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:53646->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:53646->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->45.91.116.4:4939: use of closed network connection Dec 28 12:43:24.695121270 (549h14m24.566204315s) [noise=true , address=45.91.116.4:4939 , piece_id=7ALA7BX7S7BNWJWVBHYFFI4D3L2QUGIKJT3YFXGLI3EATXKWUVVQ, seg_pos={0 0}, upload=481570]: Long tail cancellation due to optimal threshold Dec 28 12:43:24.695447462 (549h14m24.566530507s) [noise=true , address=45.91.116.11:4118 , piece_id=Q6DAYRJ5Q2XVV5OD44HLQUSR234EVNYU44DSET2LBJAVZKJQBEPQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:53180->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:53180->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53180->45.91.116.11:4118: use of closed network connection Dec 28 12:43:24.695455857 (549h14m24.566538902s) [noise=true , address=45.91.116.11:4118 , piece_id=Q6DAYRJ5Q2XVV5OD44HLQUSR234EVNYU44DSET2LBJAVZKJQBEPQ, seg_pos={0 0}, upload=481570]: Long tail cancellation due to optimal threshold Dec 28 12:43:24.695619027 (549h14m24.566702082s) [noise=true , address=45.91.116.12:4227 , piece_id=DTKN5KTHQE4K2HDOL4EMMDNRVXWE2TJE774IYOYVZDDUWC76HESQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:57268->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 12:43:24.695625350 (549h14m24.566708395s) [noise=true , address=45.91.116.12:4227 , piece_id=DTKN5KTHQE4K2HDOL4EMMDNRVXWE2TJE774IYOYVZDDUWC76HESQ, seg_pos={0 0}, upload=481570]: Long tail cancellation due to optimal threshold Dec 28 12:43:24.695641200 (549h14m24.566724255s) [seg_pos={0 0}, upload=481570]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:43:24.695713367 (549h14m24.566796402s) [upload=481570]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:43:24.695720390 (549h14m24.566803445s) [upload=481570]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:43:24.994903084 (549h14m24.865986129s) [upload=481570]: Done uploading segments... Dec 28 12:43:24.994926469 (549h14m24.866009514s) [upload=481570]: Upload finished. err: <nil> Duration: 5.768269022s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X.] duration: 4.993504601s trace: https://jaeger.dev.storj.io/trace/58701863bd9aab9e FAIL (2025-12-28 12:44:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.993533216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.........X..XX.X....XX.X.....XX..............X.X.......X] duration: 30.004290282s trace: https://jaeger.dev.storj.io/trace/b260c7acf5eda5 FAIL (2025-12-28 12:43:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: SHPKVZS3T5GPYXUOWO7AFZG2IMKLIVVF7VCFF2LTD7UE3PGJ2VLA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: ITLQDSDXS7SAEFCSDIR2LOHRDZ3TOTBOBXJ6MDIJWXZOFWKWLQ6Q) context deadline exceeded ======LOGS====== alse Dec 28 12:43:22.092912096 (549h14m21.963995131s) [upload=481570]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:43:22.092917186 (549h14m21.964000231s) [upload=481570]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:43:22.339171386 (549h14m22.210254441s) [seg_pos={0 0}, upload=481570]: Begin upload segment... Dec 28 12:43:22.339236040 (549h14m22.210319075s) [seg_pos={0 0}, upload=481570]: Done begin upload segment. Dec 28 12:43:22.339268501 (549h14m22.210351546s) [noise=true , address=145.79.227.37:4567 , piece_id=UIRTJ6KCOFKC4NQRH35SYPUGQOZKGGV3HRHKXAGXJUHJTR2TAGOA, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.339279061 (549h14m22.210362096s) [noise=true , address=145.79.227.36:4517 , piece_id=LNP7WULCMEXLLQQ5YC6C5HTYPOA33K2ZT6W5BCQTN26OQY3BIOIQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.339352160 (549h14m22.210435195s) [noise=true , address=145.79.227.39:4014 , piece_id=DIPPF7NGJQABWKAH3KQMA3UI34IODDVDZ47XOH5IPQUKI6CTY5UQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.340027235 (549h14m22.21111028s) [noise=true , address=77.90.31.19:4841 , piece_id=SHPKVZS3T5GPYXUOWO7AFZG2IMKLIVVF7VCFF2LTD7UE3PGJ2VLA, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.340041874 (549h14m22.211124929s) [noise=true , address=45.91.116.4:4939 , piece_id=7ALA7BX7S7BNWJWVBHYFFI4D3L2QUGIKJT3YFXGLI3EATXKWUVVQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.340618712 (549h14m22.211701757s) [noise=true , address=45.91.116.12:4227 , piece_id=DTKN5KTHQE4K2HDOL4EMMDNRVXWE2TJE774IYOYVZDDUWC76HESQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.340680129 (549h14m22.211763174s) [noise=true , address=45.91.116.10:4082 , piece_id=PLQEADE7EJM3XM7CJP4CZJ45Z7UQDXEXI4Z7NH32TGTAPLSJRHVQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.341129675 (549h14m22.21221271s) [noise=true , address=45.91.116.8:4045 , piece_id=DTHRCPL4INBRDDEULKI7ZL27QMRICKEDB5PLST4ARBLQ7AK4W57Q, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.341140666 (549h14m22.212223701s) [noise=true , address=145.79.227.42:4122 , piece_id=4RWNA3RY2IXYPLHGQ2TR43LKFSKIAYLCUI3MVIWU3KVTISYH4UVQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.341591614 (549h14m22.212674659s) [noise=true , address=77.90.31.21:4877 , piece_id=ITLQDSDXS7SAEFCSDIR2LOHRDZ3TOTBOBXJ6MDIJWXZOFWKWLQ6Q, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.341626571 (549h14m22.212709606s) [noise=true , address=45.91.116.11:4118 , piece_id=Q6DAYRJ5Q2XVV5OD44HLQUSR234EVNYU44DSET2LBJAVZKJQBEPQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.342223167 (549h14m22.213306212s) [noise=true , address=145.79.227.46:4603 , piece_id=LJPUYPWCLIGI5BJTXRNRUSOOOHZABB46ENFUYXUBC23E2ITQ6XIQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.342484775 (549h14m22.21356782s) [noise=true , address=145.79.227.20:4439 , piece_id=HMOXT7RVRMWI7GXHYS4N4GPRYVBNH3FIISOR2MEZY6IOWUATATUA, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.342652365 (549h14m22.21373541s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJOEUPZ2EPA7O3TFKB2IL26I7PL3ETA4SCFKG4KU77DTFLV674A, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.343146335 (549h14m22.21422937s) [noise=true , address=145.79.227.28:4726 , piece_id=LXWVUGMGSC4I4T5GN6ZRERJMCODK56KSBTIU32RTBD36RTZS4YVA, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:22.343616400 (549h14m22.214699455s) [noise=true , address=145.79.227.52:4445 , piece_id=BY5WCTBSZ7FGX73FN4UCMBTB2YQJJO2VYJC6K5QUAURUVXVPUBOQ, seg_pos={0 0}, upload=481570]: Uploading piece... Dec 28 12:43:23.963232341 (549h14m23.834315386s) [upload=481570]: Next returned nil segment Dec 28 12:43:23.963249493 (549h14m23.834332528s) [upload=481570]: Waiting for error group managing segments... Dec 28 12:43:24.343816276 (549h14m24.214899341s) [noise=true , address=145.79.227.20:4439 , piece_id=HMOXT7RVRMWI7GXHYS4N4GPRYVBNH3FIISOR2MEZY6IOWUATATUA, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.346537077 (549h14m24.217620132s) [noise=true , address=145.79.227.39:4014 , piece_id=DIPPF7NGJQABWKAH3KQMA3UI34IODDVDZ47XOH5IPQUKI6CTY5UQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.348499945 (549h14m24.21958299s) [noise=true , address=145.79.227.46:4603 , piece_id=LJPUYPWCLIGI5BJTXRNRUSOOOHZABB46ENFUYXUBC23E2ITQ6XIQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.348628870 (549h14m24.219711915s) [noise=true , address=145.79.227.37:4567 , piece_id=UIRTJ6KCOFKC4NQRH35SYPUGQOZKGGV3HRHKXAGXJUHJTR2TAGOA, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.349147528 (549h14m24.220230573s) [noise=true , address=145.79.227.15:4875 , piece_id=AWJOEUPZ2EPA7O3TFKB2IL26I7PL3ETA4SCFKG4KU77DTFLV674A, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.352771769 (549h14m24.223854804s) [noise=true , address=145.79.227.52:4445 , piece_id=BY5WCTBSZ7FGX73FN4UCMBTB2YQJJO2VYJC6K5QUAURUVXVPUBOQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.362662552 (549h14m24.233745598s) [noise=true , address=145.79.227.42:4122 , piece_id=4RWNA3RY2IXYPLHGQ2TR43LKFSKIAYLCUI3MVIWU3KVTISYH4UVQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.363249520 (549h14m24.234332555s) [noise=true , address=145.79.227.36:4517 , piece_id=LNP7WULCMEXLLQQ5YC6C5HTYPOA33K2ZT6W5BCQTN26OQY3BIOIQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.370520305 (549h14m24.24160334s) [noise=true , address=145.79.227.28:4726 , piece_id=LXWVUGMGSC4I4T5GN6ZRERJMCODK56KSBTIU32RTBD36RTZS4YVA, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.510838745 (549h14m24.38192179s) [noise=true , address=77.90.31.19:4841 , piece_id=SHPKVZS3T5GPYXUOWO7AFZG2IMKLIVVF7VCFF2LTD7UE3PGJ2VLA, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.653958389 (549h14m24.525041424s) [noise=true , address=77.90.31.21:4877 , piece_id=ITLQDSDXS7SAEFCSDIR2LOHRDZ3TOTBOBXJ6MDIJWXZOFWKWLQ6Q, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.689479327 (549h14m24.560562372s) [noise=true , address=45.91.116.8:4045 , piece_id=DTHRCPL4INBRDDEULKI7ZL27QMRICKEDB5PLST4ARBLQ7AK4W57Q, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.694874881 (549h14m24.565957926s) [noise=true , address=45.91.116.10:4082 , piece_id=PLQEADE7EJM3XM7CJP4CZJ45Z7UQDXEXI4Z7NH32TGTAPLSJRHVQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: <nil> Dec 28 12:43:24.694889939 (549h14m24.565972994s) [seg_pos={0 0}, upload=481570]: Segment reached optimal threshold of 13 pieces. Dec 28 12:43:24.695114016 (549h14m24.566197061s) [noise=true , address=45.91.116.4:4939 , piece_id=7ALA7BX7S7BNWJWVBHYFFI4D3L2QUGIKJT3YFXGLI3EATXKWUVVQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:53646->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:53646->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->45.91.116.4:4939: use of closed network connection Dec 28 12:43:24.695121270 (549h14m24.566204315s) [noise=true , address=45.91.116.4:4939 , piece_id=7ALA7BX7S7BNWJWVBHYFFI4D3L2QUGIKJT3YFXGLI3EATXKWUVVQ, seg_pos={0 0}, upload=481570]: Long tail cancellation due to optimal threshold Dec 28 12:43:24.695447462 (549h14m24.566530507s) [noise=true , address=45.91.116.11:4118 , piece_id=Q6DAYRJ5Q2XVV5OD44HLQUSR234EVNYU44DSET2LBJAVZKJQBEPQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:53180->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:53180->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53180->45.91.116.11:4118: use of closed network connection Dec 28 12:43:24.695455857 (549h14m24.566538902s) [noise=true , address=45.91.116.11:4118 , piece_id=Q6DAYRJ5Q2XVV5OD44HLQUSR234EVNYU44DSET2LBJAVZKJQBEPQ, seg_pos={0 0}, upload=481570]: Long tail cancellation due to optimal threshold Dec 28 12:43:24.695619027 (549h14m24.566702082s) [noise=true , address=45.91.116.12:4227 , piece_id=DTKN5KTHQE4K2HDOL4EMMDNRVXWE2TJE774IYOYVZDDUWC76HESQ, seg_pos={0 0}, upload=481570]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:57268->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 12:43:24.695625350 (549h14m24.566708395s) [noise=true , address=45.91.116.12:4227 , piece_id=DTKN5KTHQE4K2HDOL4EMMDNRVXWE2TJE774IYOYVZDDUWC76HESQ, seg_pos={0 0}, upload=481570]: Long tail cancellation due to optimal threshold Dec 28 12:43:24.695641200 (549h14m24.566724255s) [seg_pos={0 0}, upload=481570]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:43:24.695713367 (549h14m24.566796402s) [upload=481570]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:43:24.695720390 (549h14m24.566803445s) [upload=481570]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:43:24.994903084 (549h14m24.865986129s) [upload=481570]: Done uploading segments... Dec 28 12:43:24.994926469 (549h14m24.866009514s) [upload=481570]: Upload finished. err: <nil> Duration: 30.004318417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 39.93068738s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:43:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 39.930706766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X] duration: 5.001442434s trace: https://jaeger.dev.storj.io/trace/58701863bd9aab9e FAIL (2025-12-28 12:43:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001470417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 49.066253447s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:42:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 49.066288504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X...X.........X..XX.X....XX.X.....XX..............X.X.......] duration: 5.736818827s trace: https://jaeger.dev.storj.io/trace/787542880a31c9cd FAIL (2025-12-28 12:42:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 12:37:17.732081086 (549h8m17.603164131s) [upload=481495]: Starting upload Dec 28 12:37:17.732133236 (549h8m17.603216281s) [upload=481495]: Done starting upload Dec 28 12:37:17.732237414 (549h8m17.603320449s) [upload=481495]: Uploading segments... Dec 28 12:37:17.732351131 (549h8m17.603434176s) [upload=481495]: Got next segment. Inline: false Dec 28 12:37:17.732395775 (549h8m17.60347882s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:37:17.732401908 (549h8m17.603484943s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:37:17.981257960 (549h8m17.852341005s) [seg_pos={0 0}, upload=481495]: Begin upload segment... Dec 28 12:37:17.981304950 (549h8m17.852387985s) [seg_pos={0 0}, upload=481495]: Done begin upload segment. Dec 28 12:37:17.981345627 (549h8m17.852428672s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.981469453 (549h8m17.852552488s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982113529 (549h8m17.853196564s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982421276 (549h8m17.853504311s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982848980 (549h8m17.853932015s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983162036 (549h8m17.854245071s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983506492 (549h8m17.854589527s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983795081 (549h8m17.854878116s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.984126032 (549h8m17.855209067s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.985220756 (549h8m17.856303791s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.986684573 (549h8m17.857767618s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.987641816 (549h8m17.858724851s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.988929569 (549h8m17.860012614s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.990475301 (549h8m17.861558337s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.992038839 (549h8m17.863121874s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.993650357 (549h8m17.864733402s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:19.703954824 (549h8m19.575037869s) [upload=481495]: Next returned nil segment Dec 28 12:37:19.703979491 (549h8m19.575062536s) [upload=481495]: Waiting for error group managing segments... Dec 28 12:37:19.989096531 (549h8m19.860179566s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990424099 (549h8m19.861507134s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990832187 (549h8m19.861915222s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.991223111 (549h8m19.862306146s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.999626542 (549h8m19.870709577s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.003002049 (549h8m19.874085115s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.004071416 (549h8m19.875154461s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.011856070 (549h8m19.882939105s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.014643337 (549h8m19.885726372s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.021360428 (549h8m19.892443463s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.201187328 (549h8m20.072270363s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.266866322 (549h8m20.137949367s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276810127 (549h8m20.147893162s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276829503 (549h8m20.147912548s) [seg_pos={0 0}, upload=481495]: Segment reached optimal threshold of 13 pieces. Dec 28 12:37:20.277019766 (549h8m20.148102811s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277025807 (549h8m20.148108842s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277163780 (549h8m20.148246825s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277170062 (549h8m20.148253097s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277428885 (549h8m20.14851192s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection Dec 28 12:37:20.277448522 (549h8m20.148531567s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277530499 (549h8m20.148613544s) [seg_pos={0 0}, upload=481495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:37:20.277785203 (549h8m20.148868238s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:37:20.277817665 (549h8m20.14890072s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:37:20.567974132 (549h8m20.439057187s) [upload=481495]: Done uploading segments... Dec 28 12:37:20.567996163 (549h8m20.439079208s) [upload=481495]: Upload finished. err: <nil> Duration: 5.736847531s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...] duration: 3.530837299s trace: https://jaeger.dev.storj.io/trace/797f31129f59c234 FAIL (2025-12-28 12:42:46 UTC) orbiter warning: context deadline exceeded Duration: 3.530865341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 23.443688169s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:42:30 UTC) orbiter warning: context deadline exceeded Duration: 23.443714458s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X...X.........X..XX.X....XX.X.....XX..............X.X......] duration: 5.956241904s trace: https://jaeger.dev.storj.io/trace/787542880a31c9cd FAIL (2025-12-28 12:41:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 12:37:17.732081086 (549h8m17.603164131s) [upload=481495]: Starting upload Dec 28 12:37:17.732133236 (549h8m17.603216281s) [upload=481495]: Done starting upload Dec 28 12:37:17.732237414 (549h8m17.603320449s) [upload=481495]: Uploading segments... Dec 28 12:37:17.732351131 (549h8m17.603434176s) [upload=481495]: Got next segment. Inline: false Dec 28 12:37:17.732395775 (549h8m17.60347882s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:37:17.732401908 (549h8m17.603484943s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:37:17.981257960 (549h8m17.852341005s) [seg_pos={0 0}, upload=481495]: Begin upload segment... Dec 28 12:37:17.981304950 (549h8m17.852387985s) [seg_pos={0 0}, upload=481495]: Done begin upload segment. Dec 28 12:37:17.981345627 (549h8m17.852428672s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.981469453 (549h8m17.852552488s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982113529 (549h8m17.853196564s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982421276 (549h8m17.853504311s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982848980 (549h8m17.853932015s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983162036 (549h8m17.854245071s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983506492 (549h8m17.854589527s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983795081 (549h8m17.854878116s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.984126032 (549h8m17.855209067s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.985220756 (549h8m17.856303791s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.986684573 (549h8m17.857767618s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.987641816 (549h8m17.858724851s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.988929569 (549h8m17.860012614s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.990475301 (549h8m17.861558337s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.992038839 (549h8m17.863121874s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.993650357 (549h8m17.864733402s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:19.703954824 (549h8m19.575037869s) [upload=481495]: Next returned nil segment Dec 28 12:37:19.703979491 (549h8m19.575062536s) [upload=481495]: Waiting for error group managing segments... Dec 28 12:37:19.989096531 (549h8m19.860179566s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990424099 (549h8m19.861507134s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990832187 (549h8m19.861915222s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.991223111 (549h8m19.862306146s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.999626542 (549h8m19.870709577s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.003002049 (549h8m19.874085115s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.004071416 (549h8m19.875154461s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.011856070 (549h8m19.882939105s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.014643337 (549h8m19.885726372s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.021360428 (549h8m19.892443463s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.201187328 (549h8m20.072270363s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.266866322 (549h8m20.137949367s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276810127 (549h8m20.147893162s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276829503 (549h8m20.147912548s) [seg_pos={0 0}, upload=481495]: Segment reached optimal threshold of 13 pieces. Dec 28 12:37:20.277019766 (549h8m20.148102811s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277025807 (549h8m20.148108842s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277163780 (549h8m20.148246825s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277170062 (549h8m20.148253097s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277428885 (549h8m20.14851192s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection Dec 28 12:37:20.277448522 (549h8m20.148531567s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277530499 (549h8m20.148613544s) [seg_pos={0 0}, upload=481495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:37:20.277785203 (549h8m20.148868238s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:37:20.277817665 (549h8m20.14890072s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:37:20.567974132 (549h8m20.439057187s) [upload=481495]: Done uploading segments... Dec 28 12:37:20.567996163 (549h8m20.439079208s) [upload=481495]: Upload finished. err: <nil> Duration: 5.956263025s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [X..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X..] duration: 3.48574774s trace: https://jaeger.dev.storj.io/trace/797f31129f59c234 FAIL (2025-12-28 12:41:52 UTC) orbiter warning: context deadline exceeded Duration: 3.485773988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 42.596900679s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:41:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 42.596933261s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 14.6508991s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:41:36 UTC) orbiter warning: context deadline exceeded Duration: 14.650939277s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X.] duration: 5.00112522s trace: https://jaeger.dev.storj.io/trace/797f31129f59c234 FAIL (2025-12-28 12:41:29 UTC) orbiter warning: context deadline exceeded Duration: 5.001148374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X] duration: 5.001649858s trace: https://jaeger.dev.storj.io/trace/797f31129f59c234 FAIL (2025-12-28 12:40:58 UTC) orbiter warning: context deadline exceeded Duration: 5.001676609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X...X.........X..XX.X....XX.X.....XX..............X.X.....] duration: 5.692714873s trace: https://jaeger.dev.storj.io/trace/787542880a31c9cd FAIL (2025-12-28 12:40:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 12:37:17.732081086 (549h8m17.603164131s) [upload=481495]: Starting upload Dec 28 12:37:17.732133236 (549h8m17.603216281s) [upload=481495]: Done starting upload Dec 28 12:37:17.732237414 (549h8m17.603320449s) [upload=481495]: Uploading segments... Dec 28 12:37:17.732351131 (549h8m17.603434176s) [upload=481495]: Got next segment. Inline: false Dec 28 12:37:17.732395775 (549h8m17.60347882s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:37:17.732401908 (549h8m17.603484943s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:37:17.981257960 (549h8m17.852341005s) [seg_pos={0 0}, upload=481495]: Begin upload segment... Dec 28 12:37:17.981304950 (549h8m17.852387985s) [seg_pos={0 0}, upload=481495]: Done begin upload segment. Dec 28 12:37:17.981345627 (549h8m17.852428672s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.981469453 (549h8m17.852552488s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982113529 (549h8m17.853196564s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982421276 (549h8m17.853504311s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982848980 (549h8m17.853932015s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983162036 (549h8m17.854245071s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983506492 (549h8m17.854589527s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983795081 (549h8m17.854878116s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.984126032 (549h8m17.855209067s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.985220756 (549h8m17.856303791s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.986684573 (549h8m17.857767618s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.987641816 (549h8m17.858724851s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.988929569 (549h8m17.860012614s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.990475301 (549h8m17.861558337s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.992038839 (549h8m17.863121874s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.993650357 (549h8m17.864733402s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:19.703954824 (549h8m19.575037869s) [upload=481495]: Next returned nil segment Dec 28 12:37:19.703979491 (549h8m19.575062536s) [upload=481495]: Waiting for error group managing segments... Dec 28 12:37:19.989096531 (549h8m19.860179566s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990424099 (549h8m19.861507134s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990832187 (549h8m19.861915222s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.991223111 (549h8m19.862306146s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.999626542 (549h8m19.870709577s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.003002049 (549h8m19.874085115s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.004071416 (549h8m19.875154461s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.011856070 (549h8m19.882939105s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.014643337 (549h8m19.885726372s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.021360428 (549h8m19.892443463s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.201187328 (549h8m20.072270363s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.266866322 (549h8m20.137949367s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276810127 (549h8m20.147893162s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276829503 (549h8m20.147912548s) [seg_pos={0 0}, upload=481495]: Segment reached optimal threshold of 13 pieces. Dec 28 12:37:20.277019766 (549h8m20.148102811s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277025807 (549h8m20.148108842s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277163780 (549h8m20.148246825s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277170062 (549h8m20.148253097s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277428885 (549h8m20.14851192s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection Dec 28 12:37:20.277448522 (549h8m20.148531567s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277530499 (549h8m20.148613544s) [seg_pos={0 0}, upload=481495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:37:20.277785203 (549h8m20.148868238s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:37:20.277817665 (549h8m20.14890072s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:37:20.567974132 (549h8m20.439057187s) [upload=481495]: Done uploading segments... Dec 28 12:37:20.567996163 (549h8m20.439079208s) [upload=481495]: Upload finished. err: <nil> Duration: 5.692739722s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 12.485630236s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:40:48 UTC) orbiter warning: context deadline exceeded Duration: 12.485655743s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 40.963188251s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 40.963209493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.] duration: 3.546512446s trace: https://jaeger.dev.storj.io/trace/3ea681ad4b13cd9d FAIL (2025-12-28 12:40:08 UTC) orbiter warning: context deadline exceeded Duration: 3.546543686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 45.66061092s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:40:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 45.660708495s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 26.937046109s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:39:56 UTC) orbiter warning: context deadline exceeded Duration: 26.937073953s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X...X.........X..XX.X....XX.X.....XX..............X.X....] duration: 5.649907233s trace: https://jaeger.dev.storj.io/trace/787542880a31c9cd FAIL (2025-12-28 12:39:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 12:37:17.732081086 (549h8m17.603164131s) [upload=481495]: Starting upload Dec 28 12:37:17.732133236 (549h8m17.603216281s) [upload=481495]: Done starting upload Dec 28 12:37:17.732237414 (549h8m17.603320449s) [upload=481495]: Uploading segments... Dec 28 12:37:17.732351131 (549h8m17.603434176s) [upload=481495]: Got next segment. Inline: false Dec 28 12:37:17.732395775 (549h8m17.60347882s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:37:17.732401908 (549h8m17.603484943s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:37:17.981257960 (549h8m17.852341005s) [seg_pos={0 0}, upload=481495]: Begin upload segment... Dec 28 12:37:17.981304950 (549h8m17.852387985s) [seg_pos={0 0}, upload=481495]: Done begin upload segment. Dec 28 12:37:17.981345627 (549h8m17.852428672s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.981469453 (549h8m17.852552488s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982113529 (549h8m17.853196564s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982421276 (549h8m17.853504311s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982848980 (549h8m17.853932015s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983162036 (549h8m17.854245071s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983506492 (549h8m17.854589527s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983795081 (549h8m17.854878116s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.984126032 (549h8m17.855209067s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.985220756 (549h8m17.856303791s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.986684573 (549h8m17.857767618s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.987641816 (549h8m17.858724851s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.988929569 (549h8m17.860012614s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.990475301 (549h8m17.861558337s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.992038839 (549h8m17.863121874s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.993650357 (549h8m17.864733402s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:19.703954824 (549h8m19.575037869s) [upload=481495]: Next returned nil segment Dec 28 12:37:19.703979491 (549h8m19.575062536s) [upload=481495]: Waiting for error group managing segments... Dec 28 12:37:19.989096531 (549h8m19.860179566s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990424099 (549h8m19.861507134s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990832187 (549h8m19.861915222s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.991223111 (549h8m19.862306146s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.999626542 (549h8m19.870709577s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.003002049 (549h8m19.874085115s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.004071416 (549h8m19.875154461s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.011856070 (549h8m19.882939105s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.014643337 (549h8m19.885726372s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.021360428 (549h8m19.892443463s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.201187328 (549h8m20.072270363s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.266866322 (549h8m20.137949367s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276810127 (549h8m20.147893162s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276829503 (549h8m20.147912548s) [seg_pos={0 0}, upload=481495]: Segment reached optimal threshold of 13 pieces. Dec 28 12:37:20.277019766 (549h8m20.148102811s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277025807 (549h8m20.148108842s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277163780 (549h8m20.148246825s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277170062 (549h8m20.148253097s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277428885 (549h8m20.14851192s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection Dec 28 12:37:20.277448522 (549h8m20.148531567s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277530499 (549h8m20.148613544s) [seg_pos={0 0}, upload=481495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:37:20.277785203 (549h8m20.148868238s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:37:20.277817665 (549h8m20.14890072s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:37:20.567974132 (549h8m20.439057187s) [upload=481495]: Done uploading segments... Dec 28 12:37:20.567996163 (549h8m20.439079208s) [upload=481495]: Upload finished. err: <nil> Duration: 5.649936969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 44.813836554s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:39:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 44.81386576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX] duration: 5.001494875s trace: https://jaeger.dev.storj.io/trace/3ea681ad4b13cd9d FAIL (2025-12-28 12:39:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001520774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X...X.........X..XX.X....XX.X.....XX..............X.X...] duration: 5.804968675s trace: https://jaeger.dev.storj.io/trace/787542880a31c9cd FAIL (2025-12-28 12:38:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 12:37:17.732081086 (549h8m17.603164131s) [upload=481495]: Starting upload Dec 28 12:37:17.732133236 (549h8m17.603216281s) [upload=481495]: Done starting upload Dec 28 12:37:17.732237414 (549h8m17.603320449s) [upload=481495]: Uploading segments... Dec 28 12:37:17.732351131 (549h8m17.603434176s) [upload=481495]: Got next segment. Inline: false Dec 28 12:37:17.732395775 (549h8m17.60347882s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:37:17.732401908 (549h8m17.603484943s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:37:17.981257960 (549h8m17.852341005s) [seg_pos={0 0}, upload=481495]: Begin upload segment... Dec 28 12:37:17.981304950 (549h8m17.852387985s) [seg_pos={0 0}, upload=481495]: Done begin upload segment. Dec 28 12:37:17.981345627 (549h8m17.852428672s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.981469453 (549h8m17.852552488s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982113529 (549h8m17.853196564s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982421276 (549h8m17.853504311s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982848980 (549h8m17.853932015s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983162036 (549h8m17.854245071s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983506492 (549h8m17.854589527s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983795081 (549h8m17.854878116s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.984126032 (549h8m17.855209067s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.985220756 (549h8m17.856303791s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.986684573 (549h8m17.857767618s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.987641816 (549h8m17.858724851s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.988929569 (549h8m17.860012614s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.990475301 (549h8m17.861558337s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.992038839 (549h8m17.863121874s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.993650357 (549h8m17.864733402s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:19.703954824 (549h8m19.575037869s) [upload=481495]: Next returned nil segment Dec 28 12:37:19.703979491 (549h8m19.575062536s) [upload=481495]: Waiting for error group managing segments... Dec 28 12:37:19.989096531 (549h8m19.860179566s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990424099 (549h8m19.861507134s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990832187 (549h8m19.861915222s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.991223111 (549h8m19.862306146s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.999626542 (549h8m19.870709577s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.003002049 (549h8m19.874085115s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.004071416 (549h8m19.875154461s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.011856070 (549h8m19.882939105s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.014643337 (549h8m19.885726372s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.021360428 (549h8m19.892443463s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.201187328 (549h8m20.072270363s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.266866322 (549h8m20.137949367s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276810127 (549h8m20.147893162s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276829503 (549h8m20.147912548s) [seg_pos={0 0}, upload=481495]: Segment reached optimal threshold of 13 pieces. Dec 28 12:37:20.277019766 (549h8m20.148102811s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277025807 (549h8m20.148108842s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277163780 (549h8m20.148246825s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277170062 (549h8m20.148253097s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277428885 (549h8m20.14851192s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection Dec 28 12:37:20.277448522 (549h8m20.148531567s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277530499 (549h8m20.148613544s) [seg_pos={0 0}, upload=481495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:37:20.277785203 (549h8m20.148868238s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:37:20.277817665 (549h8m20.14890072s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:37:20.567974132 (549h8m20.439057187s) [upload=481495]: Done uploading segments... Dec 28 12:37:20.567996163 (549h8m20.439079208s) [upload=481495]: Upload finished. err: <nil> Duration: 5.804998933s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration: 24.388357817s trace: https://jaeger.dev.storj.io/trace/72d0191c7bb67437 FAIL (2025-12-28 12:38:47 UTC) orbiter warning: context deadline exceeded Duration: 24.388389075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X....X...X.........X..XX.X....XX.X.....XX..............X.X..] duration: 5.628574149s trace: https://jaeger.dev.storj.io/trace/787542880a31c9cd FAIL (2025-12-28 12:38:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 12:37:17.732081086 (549h8m17.603164131s) [upload=481495]: Starting upload Dec 28 12:37:17.732133236 (549h8m17.603216281s) [upload=481495]: Done starting upload Dec 28 12:37:17.732237414 (549h8m17.603320449s) [upload=481495]: Uploading segments... Dec 28 12:37:17.732351131 (549h8m17.603434176s) [upload=481495]: Got next segment. Inline: false Dec 28 12:37:17.732395775 (549h8m17.60347882s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:37:17.732401908 (549h8m17.603484943s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:37:17.981257960 (549h8m17.852341005s) [seg_pos={0 0}, upload=481495]: Begin upload segment... Dec 28 12:37:17.981304950 (549h8m17.852387985s) [seg_pos={0 0}, upload=481495]: Done begin upload segment. Dec 28 12:37:17.981345627 (549h8m17.852428672s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.981469453 (549h8m17.852552488s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982113529 (549h8m17.853196564s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982421276 (549h8m17.853504311s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982848980 (549h8m17.853932015s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983162036 (549h8m17.854245071s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983506492 (549h8m17.854589527s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983795081 (549h8m17.854878116s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.984126032 (549h8m17.855209067s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.985220756 (549h8m17.856303791s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.986684573 (549h8m17.857767618s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.987641816 (549h8m17.858724851s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.988929569 (549h8m17.860012614s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.990475301 (549h8m17.861558337s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.992038839 (549h8m17.863121874s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.993650357 (549h8m17.864733402s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:19.703954824 (549h8m19.575037869s) [upload=481495]: Next returned nil segment Dec 28 12:37:19.703979491 (549h8m19.575062536s) [upload=481495]: Waiting for error group managing segments... Dec 28 12:37:19.989096531 (549h8m19.860179566s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990424099 (549h8m19.861507134s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990832187 (549h8m19.861915222s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.991223111 (549h8m19.862306146s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.999626542 (549h8m19.870709577s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.003002049 (549h8m19.874085115s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.004071416 (549h8m19.875154461s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.011856070 (549h8m19.882939105s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.014643337 (549h8m19.885726372s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.021360428 (549h8m19.892443463s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.201187328 (549h8m20.072270363s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.266866322 (549h8m20.137949367s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276810127 (549h8m20.147893162s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276829503 (549h8m20.147912548s) [seg_pos={0 0}, upload=481495]: Segment reached optimal threshold of 13 pieces. Dec 28 12:37:20.277019766 (549h8m20.148102811s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277025807 (549h8m20.148108842s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277163780 (549h8m20.148246825s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277170062 (549h8m20.148253097s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277428885 (549h8m20.14851192s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection Dec 28 12:37:20.277448522 (549h8m20.148531567s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277530499 (549h8m20.148613544s) [seg_pos={0 0}, upload=481495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:37:20.277785203 (549h8m20.148868238s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:37:20.277817665 (549h8m20.14890072s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:37:20.567974132 (549h8m20.439057187s) [upload=481495]: Done uploading segments... Dec 28 12:37:20.567996163 (549h8m20.439079208s) [upload=481495]: Upload finished. err: <nil> Duration: 5.628598997s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXXXX..XXXX...XX.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..X] duration: 5.000905543s trace: https://jaeger.dev.storj.io/trace/1da16c8ef43e6eb0 FAIL (2025-12-28 12:38:38 UTC) orbiter warning: context deadline exceeded Duration: 5.000925981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 48.379354538s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:37:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 48.37938745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX....X...X.........X..XX.X....XX.X.....XX..............X.X.] duration: 6.069874982s trace: https://jaeger.dev.storj.io/trace/787542880a31c9cd FAIL (2025-12-28 12:37:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 12:37:17.732081086 (549h8m17.603164131s) [upload=481495]: Starting upload Dec 28 12:37:17.732133236 (549h8m17.603216281s) [upload=481495]: Done starting upload Dec 28 12:37:17.732237414 (549h8m17.603320449s) [upload=481495]: Uploading segments... Dec 28 12:37:17.732351131 (549h8m17.603434176s) [upload=481495]: Got next segment. Inline: false Dec 28 12:37:17.732395775 (549h8m17.60347882s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:37:17.732401908 (549h8m17.603484943s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:37:17.981257960 (549h8m17.852341005s) [seg_pos={0 0}, upload=481495]: Begin upload segment... Dec 28 12:37:17.981304950 (549h8m17.852387985s) [seg_pos={0 0}, upload=481495]: Done begin upload segment. Dec 28 12:37:17.981345627 (549h8m17.852428672s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.981469453 (549h8m17.852552488s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982113529 (549h8m17.853196564s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982421276 (549h8m17.853504311s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982848980 (549h8m17.853932015s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983162036 (549h8m17.854245071s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983506492 (549h8m17.854589527s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983795081 (549h8m17.854878116s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.984126032 (549h8m17.855209067s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.985220756 (549h8m17.856303791s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.986684573 (549h8m17.857767618s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.987641816 (549h8m17.858724851s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.988929569 (549h8m17.860012614s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.990475301 (549h8m17.861558337s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.992038839 (549h8m17.863121874s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.993650357 (549h8m17.864733402s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:19.703954824 (549h8m19.575037869s) [upload=481495]: Next returned nil segment Dec 28 12:37:19.703979491 (549h8m19.575062536s) [upload=481495]: Waiting for error group managing segments... Dec 28 12:37:19.989096531 (549h8m19.860179566s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990424099 (549h8m19.861507134s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990832187 (549h8m19.861915222s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.991223111 (549h8m19.862306146s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.999626542 (549h8m19.870709577s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.003002049 (549h8m19.874085115s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.004071416 (549h8m19.875154461s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.011856070 (549h8m19.882939105s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.014643337 (549h8m19.885726372s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.021360428 (549h8m19.892443463s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.201187328 (549h8m20.072270363s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.266866322 (549h8m20.137949367s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276810127 (549h8m20.147893162s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276829503 (549h8m20.147912548s) [seg_pos={0 0}, upload=481495]: Segment reached optimal threshold of 13 pieces. Dec 28 12:37:20.277019766 (549h8m20.148102811s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277025807 (549h8m20.148108842s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277163780 (549h8m20.148246825s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277170062 (549h8m20.148253097s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277428885 (549h8m20.14851192s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection Dec 28 12:37:20.277448522 (549h8m20.148531567s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277530499 (549h8m20.148613544s) [seg_pos={0 0}, upload=481495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:37:20.277785203 (549h8m20.148868238s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:37:20.277817665 (549h8m20.14890072s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:37:20.567974132 (549h8m20.439057187s) [upload=481495]: Done uploading segments... Dec 28 12:37:20.567996163 (549h8m20.439079208s) [upload=481495]: Upload finished. err: <nil> Duration: 6.069888117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX....X...X.........X..XX.X....XX.X.....XX..............X.X] duration: 30.002115348s trace: https://jaeger.dev.storj.io/trace/787542880a31c9cd FAIL (2025-12-28 12:37:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 12:37:17.732081086 (549h8m17.603164131s) [upload=481495]: Starting upload Dec 28 12:37:17.732133236 (549h8m17.603216281s) [upload=481495]: Done starting upload Dec 28 12:37:17.732237414 (549h8m17.603320449s) [upload=481495]: Uploading segments... Dec 28 12:37:17.732351131 (549h8m17.603434176s) [upload=481495]: Got next segment. Inline: false Dec 28 12:37:17.732395775 (549h8m17.60347882s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:37:17.732401908 (549h8m17.603484943s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:37:17.981257960 (549h8m17.852341005s) [seg_pos={0 0}, upload=481495]: Begin upload segment... Dec 28 12:37:17.981304950 (549h8m17.852387985s) [seg_pos={0 0}, upload=481495]: Done begin upload segment. Dec 28 12:37:17.981345627 (549h8m17.852428672s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.981469453 (549h8m17.852552488s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982113529 (549h8m17.853196564s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982421276 (549h8m17.853504311s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.982848980 (549h8m17.853932015s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983162036 (549h8m17.854245071s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983506492 (549h8m17.854589527s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.983795081 (549h8m17.854878116s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.984126032 (549h8m17.855209067s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.985220756 (549h8m17.856303791s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.986684573 (549h8m17.857767618s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.987641816 (549h8m17.858724851s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.988929569 (549h8m17.860012614s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.990475301 (549h8m17.861558337s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.992038839 (549h8m17.863121874s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:17.993650357 (549h8m17.864733402s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Uploading piece... Dec 28 12:37:19.703954824 (549h8m19.575037869s) [upload=481495]: Next returned nil segment Dec 28 12:37:19.703979491 (549h8m19.575062536s) [upload=481495]: Waiting for error group managing segments... Dec 28 12:37:19.989096531 (549h8m19.860179566s) [noise=true , address=145.79.227.4:4185 , piece_id=6SIUWEVDY35JSCAKGVKNMIF265NS23TILOGQUDCQGT2EKACECXUA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990424099 (549h8m19.861507134s) [noise=true , address=145.79.227.11:4731 , piece_id=CPERNRLMR3XMQ6KWECI6A6JTBRN6UF3U2OB3YU457H5MSHVL5IAQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.990832187 (549h8m19.861915222s) [noise=true , address=145.79.227.50:4374 , piece_id=FDBEKBCTXVAAMBV7YZVHLS7RM24EURUU5IEQBYX6KBUP6VZ6HEOA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.991223111 (549h8m19.862306146s) [noise=true , address=145.79.227.48:4302 , piece_id=SNKHYYUJEXF3Z3A4FQHRSEC2EQSOI5SB4KY5ASHVWQL6IPCDD42A, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:19.999626542 (549h8m19.870709577s) [noise=true , address=145.79.227.15:4876 , piece_id=MLIXQ3OUZ7XX65A7LHTPEHF76FRCU2FXVMALZD6V5HT22AMV5YJA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.003002049 (549h8m19.874085115s) [noise=true , address=145.79.227.16:4912 , piece_id=3MLXYW4JGY5ZBQZOZPZUBZVZ2VX2YXCII6Q3I347YFRVC3BJE2OA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.004071416 (549h8m19.875154461s) [noise=true , address=145.79.227.51:4409 , piece_id=5VTMEWAPMOI5RA3XZCHITJRWYNO72IE6GUPXMRWI3LMPUMTZ5S5Q, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.011856070 (549h8m19.882939105s) [noise=true , address=145.79.227.6:4259 , piece_id=VOR4WRYFUKTBBNUZWHL7RJ2EMGANEWKGNID4ISCPJPED7E4S2KIA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.014643337 (549h8m19.885726372s) [noise=true , address=145.79.227.28:4726 , piece_id=OHAECK6L7HEVBRBDHAM3GUJXYOZBA335V6SFRTUL5EPVBJPQPNDA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.021360428 (549h8m19.892443463s) [noise=true , address=145.79.227.52:4445 , piece_id=Y7MADB27YMA7ZO4YIFS6T5B5UNC2YSQTMX52WHSGT7VNYT52WDTA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.201187328 (549h8m20.072270363s) [noise=true , address=77.90.31.12:4589 , piece_id=VBDNNNHTHO3R6YHB4HOUXH7XF2AEHGXBFLQN7CEUIFGBG33BSARA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.266866322 (549h8m20.137949367s) [noise=true , address=77.90.31.11:4556 , piece_id=HOJTHGMEBQGQH4FBAC3LIZDOD6P4B6WBLRW7PKFWZ3WPTG2DTUGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276810127 (549h8m20.147893162s) [noise=true , address=77.90.31.13:4627 , piece_id=DO6JGAVVWTIV2RP4MKIJKJFYAI467KVC65KP6LWUVDXJJRZ7QCVA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: <nil> Dec 28 12:37:20.276829503 (549h8m20.147912548s) [seg_pos={0 0}, upload=481495]: Segment reached optimal threshold of 13 pieces. Dec 28 12:37:20.277019766 (549h8m20.148102811s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277025807 (549h8m20.148108842s) [noise=true , address=45.91.116.11:4119 , piece_id=5EHP5NFN6GFAUGNJD625XVWPAMA2NDBUB5ZPSNT3ID6OSXVTTLZQ, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277163780 (549h8m20.148246825s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 28 12:37:20.277170062 (549h8m20.148253097s) [noise=true , address=45.91.116.13:4156 , piece_id=3KCKCP2Z555XSCVDTWPGZMGZ5AWGA4NQAAZKLE4LGJSMD5XD2XEA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277428885 (549h8m20.14851192s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34178->77.90.31.19:4841: use of closed network connection Dec 28 12:37:20.277448522 (549h8m20.148531567s) [noise=true , address=77.90.31.19:4841 , piece_id=2XB5R6Y4MP3OUR2YNM4OZFGMONBCVK753CMVRXRS5RUQUPNAUPGA, seg_pos={0 0}, upload=481495]: Long tail cancellation due to optimal threshold Dec 28 12:37:20.277530499 (549h8m20.148613544s) [seg_pos={0 0}, upload=481495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:37:20.277785203 (549h8m20.148868238s) [upload=481495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:37:20.277817665 (549h8m20.14890072s) [upload=481495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:37:20.567974132 (549h8m20.439057187s) [upload=481495]: Done uploading segments... Dec 28 12:37:20.567996163 (549h8m20.439079208s) [upload=481495]: Upload finished. err: <nil> Duration: 30.002148731s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)