Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X........X...X..X..X.X....X.X.......XX......XX.XX....XX..] duration: 6.186837934s trace: https://jaeger.dev.storj.io/trace/450fb03ee1f76d3c FAIL (2025-12-30 12:19:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 12:17:03.583536957 (596h48m3.454620002s) [upload=523142]: Starting upload Dec 30 12:17:03.583640825 (596h48m3.45472387s) [upload=523142]: Done starting upload Dec 30 12:17:03.583695750 (596h48m3.454778795s) [upload=523142]: Uploading segments... Dec 30 12:17:03.583906541 (596h48m3.454989586s) [upload=523142]: Got next segment. Inline: false Dec 30 12:17:03.583951236 (596h48m3.455034291s) [upload=523142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:17:03.583957768 (596h48m3.455040833s) [upload=523142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:17:03.834019132 (596h48m3.705102188s) [seg_pos={0 0}, upload=523142]: Begin upload segment... Dec 30 12:17:03.834090989 (596h48m3.705174034s) [seg_pos={0 0}, upload=523142]: Done begin upload segment. Dec 30 12:17:03.834124473 (596h48m3.705207558s) [noise=true , address=145.79.227.10:4401 , piece_id=ZXDHJ6WQJ3OUK6GTH5QXIX4Y3PLTRATESH4AR247R34CGV525SKQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.834135223 (596h48m3.705218288s) [noise=true , address=77.90.31.11:4556 , piece_id=MOHKOEJHORL7RF2PLV4R4QTXXKGIJ3TSAX3PTF3666IYCIOZUSXA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.834652699 (596h48m3.705735744s) [noise=true , address=77.90.31.17:4773 , piece_id=BKEI7OZPS2EVYUHRSAHB2IEBV6KNUMFXHP7OB4ALEX7BTTAWFB4Q, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.835080122 (596h48m3.706163167s) [noise=true , address=45.91.116.10:4081 , piece_id=ACMMVGZBGT2FLNKQFDJSPIMIVWP2WXHUFUFYZAMDE47RNB7KJAVA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.835509741 (596h48m3.706592786s) [noise=true , address=145.79.227.18:4006 , piece_id=LSOZPSBSJBCRCIYM3JJTPHHURRJKZ52KXUNIGQOOTYDBZOYWHXLA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.835953486 (596h48m3.707036531s) [noise=true , address=145.79.227.52:4446 , piece_id=FNMNETQZPWTUDTC4OT72GX23PNHIP5YIHQPNSALXEQN2MD6AGKAA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.836434892 (596h48m3.707517927s) [noise=true , address=145.79.227.20:4439 , piece_id=5MP6VWMT27DNZHJKTXKT6QDIKU4KT6YOWVNXOXYMNFQQ74M73BIA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.836902253 (596h48m3.707985298s) [noise=true , address=145.79.227.11:4731 , piece_id=QYL5OVZJ6QWXGFXJAH3MDMMZAWOVYVHSI6FNIK5VCCKXZ2YVYWPQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.838619813 (596h48m3.709702858s) [noise=true , address=45.91.116.5:4298 , piece_id=MCFQO2ECAO6PYDDQFSO5JJJKAEIEOJWWUE3K7LZPHCCBR2FQO25Q, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.840400223 (596h48m3.711483268s) [noise=true , address=145.79.227.47:4267 , piece_id=CYLEJ7FGPXIQFJFMJUX2VMSM4IO4X7PITL5Y2A75UKQ4RGTXTZOA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.842441129 (596h48m3.713524175s) [noise=true , address=145.79.227.28:4726 , piece_id=YEKRB3Z4GJXNJ2BQOPARQX3256ZD42JXOIPRVABP7Q5GGLMJRVOQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.844057096 (596h48m3.715140141s) [noise=true , address=45.91.116.12:4228 , piece_id=XZ6NMI437XY6X5PXK5ZUT53GELV5CFTFVUPSD7X5GTX6GJRAJBNQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.845151701 (596h48m3.716234746s) [noise=true , address=145.79.227.26:4654 , piece_id=HAQ4U5H4OZYO3CQJZO2W6TMJH7JHG4HZXZZRIGJCWIMNWRM2DUSQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.847294641 (596h48m3.718377686s) [noise=true , address=77.90.31.19:4841 , piece_id=YGNMPVR5ANKTWFO5DRQMHDG6S235ZI6G6A5SLIEBUNX7GHHGVE2Q, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.847695184 (596h48m3.718778229s) [noise=true , address=145.79.227.7:4293 , piece_id=DK3HO4TVLDXDFNBSOWKEWZJJ6XK4SWHKLLWM6OIQDLAU34BOXSQA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.850289594 (596h48m3.721372649s) [noise=true , address=77.90.31.10:4517 , piece_id=IGLLHHM5ICRT7BABG67PQKQOTRPKLXCELQZH2PVQLKTJZHG7H4LA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:05.556810850 (596h48m5.427893895s) [upload=523142]: Next returned nil segment Dec 30 12:17:05.556829545 (596h48m5.42791259s) [upload=523142]: Waiting for error group managing segments... Dec 30 12:17:05.842746966 (596h48m5.713830021s) [noise=true , address=145.79.227.18:4006 , piece_id=LSOZPSBSJBCRCIYM3JJTPHHURRJKZ52KXUNIGQOOTYDBZOYWHXLA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.842971784 (596h48m5.714054829s) [noise=true , address=145.79.227.11:4731 , piece_id=QYL5OVZJ6QWXGFXJAH3MDMMZAWOVYVHSI6FNIK5VCCKXZ2YVYWPQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.854681602 (596h48m5.725764637s) [noise=true , address=145.79.227.28:4726 , piece_id=YEKRB3Z4GJXNJ2BQOPARQX3256ZD42JXOIPRVABP7Q5GGLMJRVOQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.856243866 (596h48m5.727326901s) [noise=true , address=145.79.227.47:4267 , piece_id=CYLEJ7FGPXIQFJFMJUX2VMSM4IO4X7PITL5Y2A75UKQ4RGTXTZOA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.864090306 (596h48m5.735173351s) [noise=true , address=145.79.227.20:4439 , piece_id=5MP6VWMT27DNZHJKTXKT6QDIKU4KT6YOWVNXOXYMNFQQ74M73BIA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.865029334 (596h48m5.736112379s) [noise=true , address=145.79.227.26:4654 , piece_id=HAQ4U5H4OZYO3CQJZO2W6TMJH7JHG4HZXZZRIGJCWIMNWRM2DUSQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.918332853 (596h48m5.789415898s) [noise=true , address=145.79.227.10:4401 , piece_id=ZXDHJ6WQJ3OUK6GTH5QXIX4Y3PLTRATESH4AR247R34CGV525SKQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.933255287 (596h48m5.804338322s) [noise=true , address=145.79.227.52:4446 , piece_id=FNMNETQZPWTUDTC4OT72GX23PNHIP5YIHQPNSALXEQN2MD6AGKAA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.101685895 (596h48m5.97276894s) [noise=true , address=77.90.31.11:4556 , piece_id=MOHKOEJHORL7RF2PLV4R4QTXXKGIJ3TSAX3PTF3666IYCIOZUSXA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.113223915 (596h48m5.98430696s) [noise=true , address=77.90.31.17:4773 , piece_id=BKEI7OZPS2EVYUHRSAHB2IEBV6KNUMFXHP7OB4ALEX7BTTAWFB4Q, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.123986018 (596h48m5.995069053s) [noise=true , address=77.90.31.10:4517 , piece_id=IGLLHHM5ICRT7BABG67PQKQOTRPKLXCELQZH2PVQLKTJZHG7H4LA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.160421977 (596h48m6.031505012s) [noise=true , address=145.79.227.7:4293 , piece_id=DK3HO4TVLDXDFNBSOWKEWZJJ6XK4SWHKLLWM6OIQDLAU34BOXSQA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.282126381 (596h48m6.153209426s) [noise=true , address=45.91.116.10:4081 , piece_id=ACMMVGZBGT2FLNKQFDJSPIMIVWP2WXHUFUFYZAMDE47RNB7KJAVA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.282152390 (596h48m6.153235425s) [seg_pos={0 0}, upload=523142]: Segment reached optimal threshold of 13 pieces. Dec 30 12:17:06.282348103 (596h48m6.153431148s) [noise=true , address=45.91.116.12:4228 , piece_id=XZ6NMI437XY6X5PXK5ZUT53GELV5CFTFVUPSD7X5GTX6GJRAJBNQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40232->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40232->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40232->45.91.116.12:4228: use of closed network connection Dec 30 12:17:06.282355767 (596h48m6.153438802s) [noise=true , address=45.91.116.12:4228 , piece_id=XZ6NMI437XY6X5PXK5ZUT53GELV5CFTFVUPSD7X5GTX6GJRAJBNQ, seg_pos={0 0}, upload=523142]: Long tail cancellation due to optimal threshold Dec 30 12:17:06.282623888 (596h48m6.153706923s) [noise=true , address=77.90.31.19:4841 , piece_id=YGNMPVR5ANKTWFO5DRQMHDG6S235ZI6G6A5SLIEBUNX7GHHGVE2Q, seg_pos={0 0}, upload=523142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 12:17:06.282630230 (596h48m6.153713265s) [noise=true , address=77.90.31.19:4841 , piece_id=YGNMPVR5ANKTWFO5DRQMHDG6S235ZI6G6A5SLIEBUNX7GHHGVE2Q, seg_pos={0 0}, upload=523142]: Long tail cancellation due to optimal threshold Dec 30 12:17:06.282765077 (596h48m6.153848122s) [noise=true , address=45.91.116.5:4298 , piece_id=MCFQO2ECAO6PYDDQFSO5JJJKAEIEOJWWUE3K7LZPHCCBR2FQO25Q, seg_pos={0 0}, upload=523142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 12:17:06.282772371 (596h48m6.153855416s) [noise=true , address=45.91.116.5:4298 , piece_id=MCFQO2ECAO6PYDDQFSO5JJJKAEIEOJWWUE3K7LZPHCCBR2FQO25Q, seg_pos={0 0}, upload=523142]: Long tail cancellation due to optimal threshold Dec 30 12:17:06.282802538 (596h48m6.153885573s) [seg_pos={0 0}, upload=523142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:17:06.282884344 (596h48m6.153967379s) [upload=523142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:17:06.282889264 (596h48m6.153972309s) [upload=523142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:17:06.582716259 (596h48m6.453799334s) [upload=523142]: Done uploading segments... Dec 30 12:17:06.582756406 (596h48m6.453839451s) [upload=523142]: Upload finished. err: <nil> Duration: 6.186866318s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..XX] duration: 5.000706431s trace: https://jaeger.dev.storj.io/trace/11122fb19eb3bc31 FAIL (2025-12-30 12:18:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000729926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X................X.................X.................] duration: 27.343630352s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:18:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 27.343655929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X.X........X...X..X..X.X....X.X.......XX......XX.XX....XX.] duration: 5.767233883s trace: https://jaeger.dev.storj.io/trace/450fb03ee1f76d3c FAIL (2025-12-30 12:18:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 12:17:03.583536957 (596h48m3.454620002s) [upload=523142]: Starting upload Dec 30 12:17:03.583640825 (596h48m3.45472387s) [upload=523142]: Done starting upload Dec 30 12:17:03.583695750 (596h48m3.454778795s) [upload=523142]: Uploading segments... Dec 30 12:17:03.583906541 (596h48m3.454989586s) [upload=523142]: Got next segment. Inline: false Dec 30 12:17:03.583951236 (596h48m3.455034291s) [upload=523142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:17:03.583957768 (596h48m3.455040833s) [upload=523142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:17:03.834019132 (596h48m3.705102188s) [seg_pos={0 0}, upload=523142]: Begin upload segment... Dec 30 12:17:03.834090989 (596h48m3.705174034s) [seg_pos={0 0}, upload=523142]: Done begin upload segment. Dec 30 12:17:03.834124473 (596h48m3.705207558s) [noise=true , address=145.79.227.10:4401 , piece_id=ZXDHJ6WQJ3OUK6GTH5QXIX4Y3PLTRATESH4AR247R34CGV525SKQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.834135223 (596h48m3.705218288s) [noise=true , address=77.90.31.11:4556 , piece_id=MOHKOEJHORL7RF2PLV4R4QTXXKGIJ3TSAX3PTF3666IYCIOZUSXA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.834652699 (596h48m3.705735744s) [noise=true , address=77.90.31.17:4773 , piece_id=BKEI7OZPS2EVYUHRSAHB2IEBV6KNUMFXHP7OB4ALEX7BTTAWFB4Q, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.835080122 (596h48m3.706163167s) [noise=true , address=45.91.116.10:4081 , piece_id=ACMMVGZBGT2FLNKQFDJSPIMIVWP2WXHUFUFYZAMDE47RNB7KJAVA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.835509741 (596h48m3.706592786s) [noise=true , address=145.79.227.18:4006 , piece_id=LSOZPSBSJBCRCIYM3JJTPHHURRJKZ52KXUNIGQOOTYDBZOYWHXLA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.835953486 (596h48m3.707036531s) [noise=true , address=145.79.227.52:4446 , piece_id=FNMNETQZPWTUDTC4OT72GX23PNHIP5YIHQPNSALXEQN2MD6AGKAA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.836434892 (596h48m3.707517927s) [noise=true , address=145.79.227.20:4439 , piece_id=5MP6VWMT27DNZHJKTXKT6QDIKU4KT6YOWVNXOXYMNFQQ74M73BIA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.836902253 (596h48m3.707985298s) [noise=true , address=145.79.227.11:4731 , piece_id=QYL5OVZJ6QWXGFXJAH3MDMMZAWOVYVHSI6FNIK5VCCKXZ2YVYWPQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.838619813 (596h48m3.709702858s) [noise=true , address=45.91.116.5:4298 , piece_id=MCFQO2ECAO6PYDDQFSO5JJJKAEIEOJWWUE3K7LZPHCCBR2FQO25Q, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.840400223 (596h48m3.711483268s) [noise=true , address=145.79.227.47:4267 , piece_id=CYLEJ7FGPXIQFJFMJUX2VMSM4IO4X7PITL5Y2A75UKQ4RGTXTZOA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.842441129 (596h48m3.713524175s) [noise=true , address=145.79.227.28:4726 , piece_id=YEKRB3Z4GJXNJ2BQOPARQX3256ZD42JXOIPRVABP7Q5GGLMJRVOQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.844057096 (596h48m3.715140141s) [noise=true , address=45.91.116.12:4228 , piece_id=XZ6NMI437XY6X5PXK5ZUT53GELV5CFTFVUPSD7X5GTX6GJRAJBNQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.845151701 (596h48m3.716234746s) [noise=true , address=145.79.227.26:4654 , piece_id=HAQ4U5H4OZYO3CQJZO2W6TMJH7JHG4HZXZZRIGJCWIMNWRM2DUSQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.847294641 (596h48m3.718377686s) [noise=true , address=77.90.31.19:4841 , piece_id=YGNMPVR5ANKTWFO5DRQMHDG6S235ZI6G6A5SLIEBUNX7GHHGVE2Q, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.847695184 (596h48m3.718778229s) [noise=true , address=145.79.227.7:4293 , piece_id=DK3HO4TVLDXDFNBSOWKEWZJJ6XK4SWHKLLWM6OIQDLAU34BOXSQA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.850289594 (596h48m3.721372649s) [noise=true , address=77.90.31.10:4517 , piece_id=IGLLHHM5ICRT7BABG67PQKQOTRPKLXCELQZH2PVQLKTJZHG7H4LA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:05.556810850 (596h48m5.427893895s) [upload=523142]: Next returned nil segment Dec 30 12:17:05.556829545 (596h48m5.42791259s) [upload=523142]: Waiting for error group managing segments... Dec 30 12:17:05.842746966 (596h48m5.713830021s) [noise=true , address=145.79.227.18:4006 , piece_id=LSOZPSBSJBCRCIYM3JJTPHHURRJKZ52KXUNIGQOOTYDBZOYWHXLA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.842971784 (596h48m5.714054829s) [noise=true , address=145.79.227.11:4731 , piece_id=QYL5OVZJ6QWXGFXJAH3MDMMZAWOVYVHSI6FNIK5VCCKXZ2YVYWPQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.854681602 (596h48m5.725764637s) [noise=true , address=145.79.227.28:4726 , piece_id=YEKRB3Z4GJXNJ2BQOPARQX3256ZD42JXOIPRVABP7Q5GGLMJRVOQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.856243866 (596h48m5.727326901s) [noise=true , address=145.79.227.47:4267 , piece_id=CYLEJ7FGPXIQFJFMJUX2VMSM4IO4X7PITL5Y2A75UKQ4RGTXTZOA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.864090306 (596h48m5.735173351s) [noise=true , address=145.79.227.20:4439 , piece_id=5MP6VWMT27DNZHJKTXKT6QDIKU4KT6YOWVNXOXYMNFQQ74M73BIA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.865029334 (596h48m5.736112379s) [noise=true , address=145.79.227.26:4654 , piece_id=HAQ4U5H4OZYO3CQJZO2W6TMJH7JHG4HZXZZRIGJCWIMNWRM2DUSQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.918332853 (596h48m5.789415898s) [noise=true , address=145.79.227.10:4401 , piece_id=ZXDHJ6WQJ3OUK6GTH5QXIX4Y3PLTRATESH4AR247R34CGV525SKQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.933255287 (596h48m5.804338322s) [noise=true , address=145.79.227.52:4446 , piece_id=FNMNETQZPWTUDTC4OT72GX23PNHIP5YIHQPNSALXEQN2MD6AGKAA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.101685895 (596h48m5.97276894s) [noise=true , address=77.90.31.11:4556 , piece_id=MOHKOEJHORL7RF2PLV4R4QTXXKGIJ3TSAX3PTF3666IYCIOZUSXA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.113223915 (596h48m5.98430696s) [noise=true , address=77.90.31.17:4773 , piece_id=BKEI7OZPS2EVYUHRSAHB2IEBV6KNUMFXHP7OB4ALEX7BTTAWFB4Q, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.123986018 (596h48m5.995069053s) [noise=true , address=77.90.31.10:4517 , piece_id=IGLLHHM5ICRT7BABG67PQKQOTRPKLXCELQZH2PVQLKTJZHG7H4LA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.160421977 (596h48m6.031505012s) [noise=true , address=145.79.227.7:4293 , piece_id=DK3HO4TVLDXDFNBSOWKEWZJJ6XK4SWHKLLWM6OIQDLAU34BOXSQA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.282126381 (596h48m6.153209426s) [noise=true , address=45.91.116.10:4081 , piece_id=ACMMVGZBGT2FLNKQFDJSPIMIVWP2WXHUFUFYZAMDE47RNB7KJAVA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.282152390 (596h48m6.153235425s) [seg_pos={0 0}, upload=523142]: Segment reached optimal threshold of 13 pieces. Dec 30 12:17:06.282348103 (596h48m6.153431148s) [noise=true , address=45.91.116.12:4228 , piece_id=XZ6NMI437XY6X5PXK5ZUT53GELV5CFTFVUPSD7X5GTX6GJRAJBNQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40232->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40232->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40232->45.91.116.12:4228: use of closed network connection Dec 30 12:17:06.282355767 (596h48m6.153438802s) [noise=true , address=45.91.116.12:4228 , piece_id=XZ6NMI437XY6X5PXK5ZUT53GELV5CFTFVUPSD7X5GTX6GJRAJBNQ, seg_pos={0 0}, upload=523142]: Long tail cancellation due to optimal threshold Dec 30 12:17:06.282623888 (596h48m6.153706923s) [noise=true , address=77.90.31.19:4841 , piece_id=YGNMPVR5ANKTWFO5DRQMHDG6S235ZI6G6A5SLIEBUNX7GHHGVE2Q, seg_pos={0 0}, upload=523142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 12:17:06.282630230 (596h48m6.153713265s) [noise=true , address=77.90.31.19:4841 , piece_id=YGNMPVR5ANKTWFO5DRQMHDG6S235ZI6G6A5SLIEBUNX7GHHGVE2Q, seg_pos={0 0}, upload=523142]: Long tail cancellation due to optimal threshold Dec 30 12:17:06.282765077 (596h48m6.153848122s) [noise=true , address=45.91.116.5:4298 , piece_id=MCFQO2ECAO6PYDDQFSO5JJJKAEIEOJWWUE3K7LZPHCCBR2FQO25Q, seg_pos={0 0}, upload=523142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 12:17:06.282772371 (596h48m6.153855416s) [noise=true , address=45.91.116.5:4298 , piece_id=MCFQO2ECAO6PYDDQFSO5JJJKAEIEOJWWUE3K7LZPHCCBR2FQO25Q, seg_pos={0 0}, upload=523142]: Long tail cancellation due to optimal threshold Dec 30 12:17:06.282802538 (596h48m6.153885573s) [seg_pos={0 0}, upload=523142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:17:06.282884344 (596h48m6.153967379s) [upload=523142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:17:06.282889264 (596h48m6.153972309s) [upload=523142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:17:06.582716259 (596h48m6.453799334s) [upload=523142]: Done uploading segments... Dec 30 12:17:06.582756406 (596h48m6.453839451s) [upload=523142]: Upload finished. err: <nil> Duration: 5.767259083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X................X.................X................] duration: 29.789434656s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:17:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 29.789449985s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X.X........X...X..X..X.X....X.X.......XX......XX.XX....XX] duration: 30.002517699s trace: https://jaeger.dev.storj.io/trace/450fb03ee1f76d3c FAIL (2025-12-30 12:17:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 12:17:03.583536957 (596h48m3.454620002s) [upload=523142]: Starting upload Dec 30 12:17:03.583640825 (596h48m3.45472387s) [upload=523142]: Done starting upload Dec 30 12:17:03.583695750 (596h48m3.454778795s) [upload=523142]: Uploading segments... Dec 30 12:17:03.583906541 (596h48m3.454989586s) [upload=523142]: Got next segment. Inline: false Dec 30 12:17:03.583951236 (596h48m3.455034291s) [upload=523142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:17:03.583957768 (596h48m3.455040833s) [upload=523142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:17:03.834019132 (596h48m3.705102188s) [seg_pos={0 0}, upload=523142]: Begin upload segment... Dec 30 12:17:03.834090989 (596h48m3.705174034s) [seg_pos={0 0}, upload=523142]: Done begin upload segment. Dec 30 12:17:03.834124473 (596h48m3.705207558s) [noise=true , address=145.79.227.10:4401 , piece_id=ZXDHJ6WQJ3OUK6GTH5QXIX4Y3PLTRATESH4AR247R34CGV525SKQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.834135223 (596h48m3.705218288s) [noise=true , address=77.90.31.11:4556 , piece_id=MOHKOEJHORL7RF2PLV4R4QTXXKGIJ3TSAX3PTF3666IYCIOZUSXA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.834652699 (596h48m3.705735744s) [noise=true , address=77.90.31.17:4773 , piece_id=BKEI7OZPS2EVYUHRSAHB2IEBV6KNUMFXHP7OB4ALEX7BTTAWFB4Q, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.835080122 (596h48m3.706163167s) [noise=true , address=45.91.116.10:4081 , piece_id=ACMMVGZBGT2FLNKQFDJSPIMIVWP2WXHUFUFYZAMDE47RNB7KJAVA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.835509741 (596h48m3.706592786s) [noise=true , address=145.79.227.18:4006 , piece_id=LSOZPSBSJBCRCIYM3JJTPHHURRJKZ52KXUNIGQOOTYDBZOYWHXLA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.835953486 (596h48m3.707036531s) [noise=true , address=145.79.227.52:4446 , piece_id=FNMNETQZPWTUDTC4OT72GX23PNHIP5YIHQPNSALXEQN2MD6AGKAA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.836434892 (596h48m3.707517927s) [noise=true , address=145.79.227.20:4439 , piece_id=5MP6VWMT27DNZHJKTXKT6QDIKU4KT6YOWVNXOXYMNFQQ74M73BIA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.836902253 (596h48m3.707985298s) [noise=true , address=145.79.227.11:4731 , piece_id=QYL5OVZJ6QWXGFXJAH3MDMMZAWOVYVHSI6FNIK5VCCKXZ2YVYWPQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.838619813 (596h48m3.709702858s) [noise=true , address=45.91.116.5:4298 , piece_id=MCFQO2ECAO6PYDDQFSO5JJJKAEIEOJWWUE3K7LZPHCCBR2FQO25Q, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.840400223 (596h48m3.711483268s) [noise=true , address=145.79.227.47:4267 , piece_id=CYLEJ7FGPXIQFJFMJUX2VMSM4IO4X7PITL5Y2A75UKQ4RGTXTZOA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.842441129 (596h48m3.713524175s) [noise=true , address=145.79.227.28:4726 , piece_id=YEKRB3Z4GJXNJ2BQOPARQX3256ZD42JXOIPRVABP7Q5GGLMJRVOQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.844057096 (596h48m3.715140141s) [noise=true , address=45.91.116.12:4228 , piece_id=XZ6NMI437XY6X5PXK5ZUT53GELV5CFTFVUPSD7X5GTX6GJRAJBNQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.845151701 (596h48m3.716234746s) [noise=true , address=145.79.227.26:4654 , piece_id=HAQ4U5H4OZYO3CQJZO2W6TMJH7JHG4HZXZZRIGJCWIMNWRM2DUSQ, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.847294641 (596h48m3.718377686s) [noise=true , address=77.90.31.19:4841 , piece_id=YGNMPVR5ANKTWFO5DRQMHDG6S235ZI6G6A5SLIEBUNX7GHHGVE2Q, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.847695184 (596h48m3.718778229s) [noise=true , address=145.79.227.7:4293 , piece_id=DK3HO4TVLDXDFNBSOWKEWZJJ6XK4SWHKLLWM6OIQDLAU34BOXSQA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:03.850289594 (596h48m3.721372649s) [noise=true , address=77.90.31.10:4517 , piece_id=IGLLHHM5ICRT7BABG67PQKQOTRPKLXCELQZH2PVQLKTJZHG7H4LA, seg_pos={0 0}, upload=523142]: Uploading piece... Dec 30 12:17:05.556810850 (596h48m5.427893895s) [upload=523142]: Next returned nil segment Dec 30 12:17:05.556829545 (596h48m5.42791259s) [upload=523142]: Waiting for error group managing segments... Dec 30 12:17:05.842746966 (596h48m5.713830021s) [noise=true , address=145.79.227.18:4006 , piece_id=LSOZPSBSJBCRCIYM3JJTPHHURRJKZ52KXUNIGQOOTYDBZOYWHXLA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.842971784 (596h48m5.714054829s) [noise=true , address=145.79.227.11:4731 , piece_id=QYL5OVZJ6QWXGFXJAH3MDMMZAWOVYVHSI6FNIK5VCCKXZ2YVYWPQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.854681602 (596h48m5.725764637s) [noise=true , address=145.79.227.28:4726 , piece_id=YEKRB3Z4GJXNJ2BQOPARQX3256ZD42JXOIPRVABP7Q5GGLMJRVOQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.856243866 (596h48m5.727326901s) [noise=true , address=145.79.227.47:4267 , piece_id=CYLEJ7FGPXIQFJFMJUX2VMSM4IO4X7PITL5Y2A75UKQ4RGTXTZOA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.864090306 (596h48m5.735173351s) [noise=true , address=145.79.227.20:4439 , piece_id=5MP6VWMT27DNZHJKTXKT6QDIKU4KT6YOWVNXOXYMNFQQ74M73BIA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.865029334 (596h48m5.736112379s) [noise=true , address=145.79.227.26:4654 , piece_id=HAQ4U5H4OZYO3CQJZO2W6TMJH7JHG4HZXZZRIGJCWIMNWRM2DUSQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.918332853 (596h48m5.789415898s) [noise=true , address=145.79.227.10:4401 , piece_id=ZXDHJ6WQJ3OUK6GTH5QXIX4Y3PLTRATESH4AR247R34CGV525SKQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:05.933255287 (596h48m5.804338322s) [noise=true , address=145.79.227.52:4446 , piece_id=FNMNETQZPWTUDTC4OT72GX23PNHIP5YIHQPNSALXEQN2MD6AGKAA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.101685895 (596h48m5.97276894s) [noise=true , address=77.90.31.11:4556 , piece_id=MOHKOEJHORL7RF2PLV4R4QTXXKGIJ3TSAX3PTF3666IYCIOZUSXA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.113223915 (596h48m5.98430696s) [noise=true , address=77.90.31.17:4773 , piece_id=BKEI7OZPS2EVYUHRSAHB2IEBV6KNUMFXHP7OB4ALEX7BTTAWFB4Q, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.123986018 (596h48m5.995069053s) [noise=true , address=77.90.31.10:4517 , piece_id=IGLLHHM5ICRT7BABG67PQKQOTRPKLXCELQZH2PVQLKTJZHG7H4LA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.160421977 (596h48m6.031505012s) [noise=true , address=145.79.227.7:4293 , piece_id=DK3HO4TVLDXDFNBSOWKEWZJJ6XK4SWHKLLWM6OIQDLAU34BOXSQA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.282126381 (596h48m6.153209426s) [noise=true , address=45.91.116.10:4081 , piece_id=ACMMVGZBGT2FLNKQFDJSPIMIVWP2WXHUFUFYZAMDE47RNB7KJAVA, seg_pos={0 0}, upload=523142]: Done uploading piece. err: <nil> Dec 30 12:17:06.282152390 (596h48m6.153235425s) [seg_pos={0 0}, upload=523142]: Segment reached optimal threshold of 13 pieces. Dec 30 12:17:06.282348103 (596h48m6.153431148s) [noise=true , address=45.91.116.12:4228 , piece_id=XZ6NMI437XY6X5PXK5ZUT53GELV5CFTFVUPSD7X5GTX6GJRAJBNQ, seg_pos={0 0}, upload=523142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40232->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40232->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40232->45.91.116.12:4228: use of closed network connection Dec 30 12:17:06.282355767 (596h48m6.153438802s) [noise=true , address=45.91.116.12:4228 , piece_id=XZ6NMI437XY6X5PXK5ZUT53GELV5CFTFVUPSD7X5GTX6GJRAJBNQ, seg_pos={0 0}, upload=523142]: Long tail cancellation due to optimal threshold Dec 30 12:17:06.282623888 (596h48m6.153706923s) [noise=true , address=77.90.31.19:4841 , piece_id=YGNMPVR5ANKTWFO5DRQMHDG6S235ZI6G6A5SLIEBUNX7GHHGVE2Q, seg_pos={0 0}, upload=523142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 12:17:06.282630230 (596h48m6.153713265s) [noise=true , address=77.90.31.19:4841 , piece_id=YGNMPVR5ANKTWFO5DRQMHDG6S235ZI6G6A5SLIEBUNX7GHHGVE2Q, seg_pos={0 0}, upload=523142]: Long tail cancellation due to optimal threshold Dec 30 12:17:06.282765077 (596h48m6.153848122s) [noise=true , address=45.91.116.5:4298 , piece_id=MCFQO2ECAO6PYDDQFSO5JJJKAEIEOJWWUE3K7LZPHCCBR2FQO25Q, seg_pos={0 0}, upload=523142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 12:17:06.282772371 (596h48m6.153855416s) [noise=true , address=45.91.116.5:4298 , piece_id=MCFQO2ECAO6PYDDQFSO5JJJKAEIEOJWWUE3K7LZPHCCBR2FQO25Q, seg_pos={0 0}, upload=523142]: Long tail cancellation due to optimal threshold Dec 30 12:17:06.282802538 (596h48m6.153885573s) [seg_pos={0 0}, upload=523142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:17:06.282884344 (596h48m6.153967379s) [upload=523142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:17:06.282889264 (596h48m6.153972309s) [upload=523142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:17:06.582716259 (596h48m6.453799334s) [upload=523142]: Done uploading segments... Dec 30 12:17:06.582756406 (596h48m6.453839451s) [upload=523142]: Upload finished. err: <nil> Duration: 30.002527287s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..X] duration: 5.001383723s trace: https://jaeger.dev.storj.io/trace/63b4d6ef3c26ebd9 FAIL (2025-12-30 12:17:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001401618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X........X................X.................X...............] duration: 45.309132063s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:17:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 45.309159596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X..] duration: 5.001449688s trace: https://jaeger.dev.storj.io/trace/70b9363b2fdebb5 FAIL (2025-12-30 12:17:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001479967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X.X.X........X...X..X..X.X....X.X.......XX......XX.XX....X] duration: 30.003758139s trace: https://jaeger.dev.storj.io/trace/6314cfe4f22ee234 FAIL (2025-12-30 12:17:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 12:16:33.327031266 (596h47m33.198114301s) [upload=523132]: Starting upload Dec 30 12:16:33.327107260 (596h47m33.198190296s) [upload=523132]: Done starting upload Dec 30 12:16:33.327311880 (596h47m33.198394915s) [upload=523132]: Uploading segments... Dec 30 12:16:33.327373257 (596h47m33.198456312s) [upload=523132]: Got next segment. Inline: false Dec 30 12:16:33.327402913 (596h47m33.198485948s) [upload=523132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:16:33.327407002 (596h47m33.198490047s) [upload=523132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:16:33.571115541 (596h47m33.442198576s) [seg_pos={0 0}, upload=523132]: Begin upload segment... Dec 30 12:16:33.571166278 (596h47m33.442249323s) [seg_pos={0 0}, upload=523132]: Done begin upload segment. Dec 30 12:16:33.571201285 (596h47m33.44228432s) [noise=true , address=145.79.227.27:4691 , piece_id=DHG63MVH2QK4LYS56X2HAH2P7SFJCALE7AYUDOQZDKDQUZ6PN37Q, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.571382129 (596h47m33.442465184s) [noise=true , address=145.79.227.11:4731 , piece_id=S5EKMOM7SY4QG63OCNM5AEFDRK35HEH7K6424I25ZRZRQ7SIQEPQ, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.571962113 (596h47m33.443045158s) [noise=true , address=77.90.31.21:4880 , piece_id=M7OGAPTXR2QH64QRLUIPU27A22NFD2SEIVP4ZY4NW6DETFSYS4MA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.572381161 (596h47m33.443464206s) [noise=true , address=145.79.227.42:4123 , piece_id=LXYO5KST6QPY3A2EXHVKQHRNWJ2Z3WDXXHDKSHPWZ656ZA367OGA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.572833994 (596h47m33.443917039s) [noise=true , address=145.79.227.51:4410 , piece_id=5DSVXQLE3RJQQ2YECZ7B7OZUJB23MEB6FQ3TF7YFOIZV74CKR2JQ, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.573262169 (596h47m33.444345204s) [noise=true , address=145.79.227.49:4337 , piece_id=QTMIQZ3CK3OJGQ2PPPZ4SPMCQ6FUR5FJ5G7F2OHA2Z5ZJRVPYDSQ, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.573999763 (596h47m33.445082808s) [noise=true , address=145.79.227.17:4972 , piece_id=3NK3EQYQPJTHJEEOZ3N4EDBTLTKINNOSPLIGA4VQLDX5H2ZXKSUA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.574472464 (596h47m33.445555499s) [noise=true , address=145.79.227.43:4158 , piece_id=KRGN7XZUQSTTETZMKIKZHJQ4A6BSDAA5EY2SQWVCWVJ4FCZXA3SA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.575790633 (596h47m33.446873678s) [noise=true , address=77.90.31.13:4626 , piece_id=RMK225HJNIQOOWFI5OBAIUNQFLVZDQFM7RUY5EZ6NZ2ECM3TPREA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.576038004 (596h47m33.447121049s) [noise=true , address=145.79.227.6:4258 , piece_id=LE4P4X5ZCVPP5GRRYU7EDBXDDK2DTCYHNC3UDKKHK52CEJYFLD2A, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.577391862 (596h47m33.448474897s) [noise=true , address=145.79.227.15:4875 , piece_id=AALO3GOQQ2PCKEPOKNEDFJA2N2JELMMXICYZ67TZNIDHQ42DFNUA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.581338306 (596h47m33.452421341s) [noise=true , address=145.79.227.29:4943 , piece_id=OCQ6BH3PSRY2IH2VD4FTX2EH5YOGWKILK3SRUZASFMED4J4UULLA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.585472298 (596h47m33.456555343s) [noise=true , address=45.91.116.14:4189 , piece_id=LLI3PQ4KLGO7K2IOZUJOGO72BC3XQLUVQM5JVUXZBWN3MQE5YVVA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.587337390 (596h47m33.458420435s) [noise=true , address=77.90.31.11:4556 , piece_id=HPUS3WZ4MYBGZ4ZRMAUZOKOAKQQYQFJL2OA5YVVTRFY6JEENBM7A, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.590429308 (596h47m33.461512343s) [noise=true , address=145.79.227.36:4517 , piece_id=3OEO2UGCKC7MIX22MJ2WEHTMKCINSOXP4X65JBZN4ELXOTGZZLEA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:33.593277871 (596h47m33.464360946s) [noise=true , address=45.91.116.11:4117 , piece_id=OEX4C3D2E3CCIB5GDQMTCK6PSQIZCUHX4HBQF7GH7CGSBNSIYUPA, seg_pos={0 0}, upload=523132]: Uploading piece... Dec 30 12:16:35.280297541 (596h47m35.151380586s) [upload=523132]: Next returned nil segment Dec 30 12:16:35.280410145 (596h47m35.15149321s) [upload=523132]: Waiting for error group managing segments... Dec 30 12:16:35.569833982 (596h47m35.440917017s) [noise=true , address=145.79.227.6:4258 , piece_id=LE4P4X5ZCVPP5GRRYU7EDBXDDK2DTCYHNC3UDKKHK52CEJYFLD2A, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:35.572080861 (596h47m35.443163896s) [noise=true , address=145.79.227.15:4875 , piece_id=AALO3GOQQ2PCKEPOKNEDFJA2N2JELMMXICYZ67TZNIDHQ42DFNUA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:35.578229318 (596h47m35.449312353s) [noise=true , address=145.79.227.27:4691 , piece_id=DHG63MVH2QK4LYS56X2HAH2P7SFJCALE7AYUDOQZDKDQUZ6PN37Q, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:35.587302515 (596h47m35.45838556s) [noise=true , address=145.79.227.51:4410 , piece_id=5DSVXQLE3RJQQ2YECZ7B7OZUJB23MEB6FQ3TF7YFOIZV74CKR2JQ, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:35.593299655 (596h47m35.4643827s) [noise=true , address=145.79.227.11:4731 , piece_id=S5EKMOM7SY4QG63OCNM5AEFDRK35HEH7K6424I25ZRZRQ7SIQEPQ, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:35.602728770 (596h47m35.473811805s) [noise=true , address=145.79.227.42:4123 , piece_id=LXYO5KST6QPY3A2EXHVKQHRNWJ2Z3WDXXHDKSHPWZ656ZA367OGA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:35.611879824 (596h47m35.482962859s) [noise=true , address=145.79.227.36:4517 , piece_id=3OEO2UGCKC7MIX22MJ2WEHTMKCINSOXP4X65JBZN4ELXOTGZZLEA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:35.641989939 (596h47m35.513072984s) [noise=true , address=145.79.227.17:4972 , piece_id=3NK3EQYQPJTHJEEOZ3N4EDBTLTKINNOSPLIGA4VQLDX5H2ZXKSUA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:35.765067958 (596h47m35.636151003s) [noise=true , address=77.90.31.21:4880 , piece_id=M7OGAPTXR2QH64QRLUIPU27A22NFD2SEIVP4ZY4NW6DETFSYS4MA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:35.916827160 (596h47m35.787910205s) [noise=true , address=77.90.31.11:4556 , piece_id=HPUS3WZ4MYBGZ4ZRMAUZOKOAKQQYQFJL2OA5YVVTRFY6JEENBM7A, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:36.022895130 (596h47m35.893978165s) [noise=true , address=45.91.116.14:4189 , piece_id=LLI3PQ4KLGO7K2IOZUJOGO72BC3XQLUVQM5JVUXZBWN3MQE5YVVA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:36.515348015 (596h47m36.38643106s) [noise=true , address=77.90.31.13:4626 , piece_id=RMK225HJNIQOOWFI5OBAIUNQFLVZDQFM7RUY5EZ6NZ2ECM3TPREA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:36.630470184 (596h47m36.501553229s) [noise=true , address=45.91.116.11:4117 , piece_id=OEX4C3D2E3CCIB5GDQMTCK6PSQIZCUHX4HBQF7GH7CGSBNSIYUPA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: <nil> Dec 30 12:16:36.630493258 (596h47m36.501576303s) [seg_pos={0 0}, upload=523132]: Segment reached optimal threshold of 13 pieces. Dec 30 12:16:36.630639877 (596h47m36.501722912s) [noise=true , address=145.79.227.43:4158 , piece_id=KRGN7XZUQSTTETZMKIKZHJQ4A6BSDAA5EY2SQWVCWVJ4FCZXA3SA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 12:16:36.630648654 (596h47m36.501731689s) [noise=true , address=145.79.227.43:4158 , piece_id=KRGN7XZUQSTTETZMKIKZHJQ4A6BSDAA5EY2SQWVCWVJ4FCZXA3SA, seg_pos={0 0}, upload=523132]: Long tail cancellation due to optimal threshold Dec 30 12:16:36.630875526 (596h47m36.501958571s) [noise=true , address=145.79.227.49:4337 , piece_id=QTMIQZ3CK3OJGQ2PPPZ4SPMCQ6FUR5FJ5G7F2OHA2Z5ZJRVPYDSQ, seg_pos={0 0}, upload=523132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 30 12:16:36.630881207 (596h47m36.501964252s) [noise=true , address=145.79.227.49:4337 , piece_id=QTMIQZ3CK3OJGQ2PPPZ4SPMCQ6FUR5FJ5G7F2OHA2Z5ZJRVPYDSQ, seg_pos={0 0}, upload=523132]: Long tail cancellation due to optimal threshold Dec 30 12:16:36.630992328 (596h47m36.502075363s) [noise=true , address=145.79.227.29:4943 , piece_id=OCQ6BH3PSRY2IH2VD4FTX2EH5YOGWKILK3SRUZASFMED4J4UULLA, seg_pos={0 0}, upload=523132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 12:16:36.630997528 (596h47m36.502080563s) [noise=true , address=145.79.227.29:4943 , piece_id=OCQ6BH3PSRY2IH2VD4FTX2EH5YOGWKILK3SRUZASFMED4J4UULLA, seg_pos={0 0}, upload=523132]: Long tail cancellation due to optimal threshold Dec 30 12:16:36.631011425 (596h47m36.50209446s) [seg_pos={0 0}, upload=523132]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:16:36.631082991 (596h47m36.502166036s) [upload=523132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:16:36.631090385 (596h47m36.50217342s) [upload=523132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:16:36.921007321 (596h47m36.792090356s) [upload=523132]: Done uploading segments... Dec 30 12:16:36.921030174 (596h47m36.79211321s) [upload=523132]: Upload finished. err: <nil> Duration: 30.0037803s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X.] duration: 3.797278595s trace: https://jaeger.dev.storj.io/trace/70b9363b2fdebb5 FAIL (2025-12-30 12:16:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.797304985s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X........X................X.................X..............] duration: 32.296734185s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:15:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 32.296753442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.X.X.X........X...X..X..X.X....X.X.......XX......XX.XX....] duration: 5.704477012s trace: https://jaeger.dev.storj.io/trace/3773befad05e210c FAIL (2025-12-30 12:15:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 12:12:32.084099774 (596h43m31.955182819s) [upload=523077]: Starting upload Dec 30 12:12:32.084184265 (596h43m31.955267311s) [upload=523077]: Done starting upload Dec 30 12:12:32.084298102 (596h43m31.955381157s) [upload=523077]: Uploading segments... Dec 30 12:12:32.084460662 (596h43m31.955543697s) [upload=523077]: Got next segment. Inline: false Dec 30 12:12:32.084501960 (596h43m31.955585005s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:12:32.084507891 (596h43m31.955590936s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:12:32.329735714 (596h43m32.200818759s) [seg_pos={0 0}, upload=523077]: Begin upload segment... Dec 30 12:12:32.329821857 (596h43m32.200904903s) [seg_pos={0 0}, upload=523077]: Done begin upload segment. Dec 30 12:12:32.329867044 (596h43m32.200950099s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.329876352 (596h43m32.200959397s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330015867 (596h43m32.201098912s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330722923 (596h43m32.201805968s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330740678 (596h43m32.201823713s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331188230 (596h43m32.202271275s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331547804 (596h43m32.202630839s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331658455 (596h43m32.2027415s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332112069 (596h43m32.203195114s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332129633 (596h43m32.203212678s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332567826 (596h43m32.203650871s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332952449 (596h43m32.204035494s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333361859 (596h43m32.204444894s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333574864 (596h43m32.204657919s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334195897 (596h43m32.205278932s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334819795 (596h43m32.20590284s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:34.036767632 (596h43m33.907850677s) [upload=523077]: Next returned nil segment Dec 30 12:12:34.036801066 (596h43m33.907884101s) [upload=523077]: Waiting for error group managing segments... Dec 30 12:12:34.320104540 (596h43m34.191187575s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322221491 (596h43m34.193304526s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322663061 (596h43m34.193746106s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.329138792 (596h43m34.200221827s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.333089725 (596h43m34.20417277s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.336954995 (596h43m34.20803804s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.396856828 (596h43m34.267939873s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.403783257 (596h43m34.274866302s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.406970076 (596h43m34.278053121s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.412949751 (596h43m34.284032786s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.489634171 (596h43m34.360717206s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.669091115 (596h43m34.54017415s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681055958 (596h43m34.552139003s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681077820 (596h43m34.552160855s) [seg_pos={0 0}, upload=523077]: Segment reached optimal threshold of 13 pieces. Dec 30 12:12:34.681267250 (596h43m34.552350295s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection Dec 30 12:12:34.681277270 (596h43m34.552360325s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681532305 (596h43m34.55261534s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681548355 (596h43m34.55263139s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681568033 (596h43m34.552651068s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681575217 (596h43m34.552658252s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681591338 (596h43m34.552674383s) [seg_pos={0 0}, upload=523077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:12:34.681756532 (596h43m34.552839607s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:12:34.681764688 (596h43m34.552847733s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:12:34.983065131 (596h43m34.854148166s) [upload=523077]: Done uploading segments... Dec 30 12:12:34.983082584 (596h43m34.854165619s) [upload=523077]: Upload finished. err: <nil> Duration: 5.704502049s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..X] duration: 5.001261677s trace: https://jaeger.dev.storj.io/trace/70b9363b2fdebb5 FAIL (2025-12-30 12:15:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001280131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X........X................X.................X.............] duration: 36.424719601s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:15:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 36.424746932s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X..] duration: 5.000827456s trace: https://jaeger.dev.storj.io/trace/5e76f82f88d96f3 FAIL (2025-12-30 12:15:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000854338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX.X.X.X........X...X..X..X.X....X.X.......XX......XX.XX...] duration: 5.777621333s trace: https://jaeger.dev.storj.io/trace/3773befad05e210c FAIL (2025-12-30 12:15:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 12:12:32.084099774 (596h43m31.955182819s) [upload=523077]: Starting upload Dec 30 12:12:32.084184265 (596h43m31.955267311s) [upload=523077]: Done starting upload Dec 30 12:12:32.084298102 (596h43m31.955381157s) [upload=523077]: Uploading segments... Dec 30 12:12:32.084460662 (596h43m31.955543697s) [upload=523077]: Got next segment. Inline: false Dec 30 12:12:32.084501960 (596h43m31.955585005s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:12:32.084507891 (596h43m31.955590936s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:12:32.329735714 (596h43m32.200818759s) [seg_pos={0 0}, upload=523077]: Begin upload segment... Dec 30 12:12:32.329821857 (596h43m32.200904903s) [seg_pos={0 0}, upload=523077]: Done begin upload segment. Dec 30 12:12:32.329867044 (596h43m32.200950099s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.329876352 (596h43m32.200959397s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330015867 (596h43m32.201098912s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330722923 (596h43m32.201805968s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330740678 (596h43m32.201823713s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331188230 (596h43m32.202271275s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331547804 (596h43m32.202630839s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331658455 (596h43m32.2027415s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332112069 (596h43m32.203195114s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332129633 (596h43m32.203212678s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332567826 (596h43m32.203650871s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332952449 (596h43m32.204035494s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333361859 (596h43m32.204444894s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333574864 (596h43m32.204657919s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334195897 (596h43m32.205278932s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334819795 (596h43m32.20590284s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:34.036767632 (596h43m33.907850677s) [upload=523077]: Next returned nil segment Dec 30 12:12:34.036801066 (596h43m33.907884101s) [upload=523077]: Waiting for error group managing segments... Dec 30 12:12:34.320104540 (596h43m34.191187575s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322221491 (596h43m34.193304526s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322663061 (596h43m34.193746106s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.329138792 (596h43m34.200221827s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.333089725 (596h43m34.20417277s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.336954995 (596h43m34.20803804s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.396856828 (596h43m34.267939873s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.403783257 (596h43m34.274866302s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.406970076 (596h43m34.278053121s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.412949751 (596h43m34.284032786s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.489634171 (596h43m34.360717206s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.669091115 (596h43m34.54017415s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681055958 (596h43m34.552139003s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681077820 (596h43m34.552160855s) [seg_pos={0 0}, upload=523077]: Segment reached optimal threshold of 13 pieces. Dec 30 12:12:34.681267250 (596h43m34.552350295s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection Dec 30 12:12:34.681277270 (596h43m34.552360325s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681532305 (596h43m34.55261534s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681548355 (596h43m34.55263139s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681568033 (596h43m34.552651068s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681575217 (596h43m34.552658252s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681591338 (596h43m34.552674383s) [seg_pos={0 0}, upload=523077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:12:34.681756532 (596h43m34.552839607s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:12:34.681764688 (596h43m34.552847733s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:12:34.983065131 (596h43m34.854148166s) [upload=523077]: Done uploading segments... Dec 30 12:12:34.983082584 (596h43m34.854165619s) [upload=523077]: Upload finished. err: <nil> Duration: 5.777642011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X.] duration: 3.569599408s trace: https://jaeger.dev.storj.io/trace/5e76f82f88d96f3 FAIL (2025-12-30 12:14:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569624387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..X] duration: 5.001921256s trace: https://jaeger.dev.storj.io/trace/5e76f82f88d96f3 FAIL (2025-12-30 12:14:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001944261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX.X.X.X........X...X..X..X.X....X.X.......XX......XX.XX..] duration: 29.45391042s trace: https://jaeger.dev.storj.io/trace/3773befad05e210c FAIL (2025-12-30 12:14:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 12:12:32.084099774 (596h43m31.955182819s) [upload=523077]: Starting upload Dec 30 12:12:32.084184265 (596h43m31.955267311s) [upload=523077]: Done starting upload Dec 30 12:12:32.084298102 (596h43m31.955381157s) [upload=523077]: Uploading segments... Dec 30 12:12:32.084460662 (596h43m31.955543697s) [upload=523077]: Got next segment. Inline: false Dec 30 12:12:32.084501960 (596h43m31.955585005s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:12:32.084507891 (596h43m31.955590936s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:12:32.329735714 (596h43m32.200818759s) [seg_pos={0 0}, upload=523077]: Begin upload segment... Dec 30 12:12:32.329821857 (596h43m32.200904903s) [seg_pos={0 0}, upload=523077]: Done begin upload segment. Dec 30 12:12:32.329867044 (596h43m32.200950099s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.329876352 (596h43m32.200959397s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330015867 (596h43m32.201098912s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330722923 (596h43m32.201805968s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330740678 (596h43m32.201823713s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331188230 (596h43m32.202271275s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331547804 (596h43m32.202630839s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331658455 (596h43m32.2027415s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332112069 (596h43m32.203195114s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332129633 (596h43m32.203212678s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332567826 (596h43m32.203650871s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332952449 (596h43m32.204035494s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333361859 (596h43m32.204444894s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333574864 (596h43m32.204657919s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334195897 (596h43m32.205278932s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334819795 (596h43m32.20590284s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:34.036767632 (596h43m33.907850677s) [upload=523077]: Next returned nil segment Dec 30 12:12:34.036801066 (596h43m33.907884101s) [upload=523077]: Waiting for error group managing segments... Dec 30 12:12:34.320104540 (596h43m34.191187575s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322221491 (596h43m34.193304526s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322663061 (596h43m34.193746106s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.329138792 (596h43m34.200221827s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.333089725 (596h43m34.20417277s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.336954995 (596h43m34.20803804s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.396856828 (596h43m34.267939873s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.403783257 (596h43m34.274866302s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.406970076 (596h43m34.278053121s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.412949751 (596h43m34.284032786s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.489634171 (596h43m34.360717206s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.669091115 (596h43m34.54017415s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681055958 (596h43m34.552139003s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681077820 (596h43m34.552160855s) [seg_pos={0 0}, upload=523077]: Segment reached optimal threshold of 13 pieces. Dec 30 12:12:34.681267250 (596h43m34.552350295s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection Dec 30 12:12:34.681277270 (596h43m34.552360325s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681532305 (596h43m34.55261534s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681548355 (596h43m34.55263139s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681568033 (596h43m34.552651068s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681575217 (596h43m34.552658252s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681591338 (596h43m34.552674383s) [seg_pos={0 0}, upload=523077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:12:34.681756532 (596h43m34.552839607s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:12:34.681764688 (596h43m34.552847733s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:12:34.983065131 (596h43m34.854148166s) [upload=523077]: Done uploading segments... Dec 30 12:12:34.983082584 (596h43m34.854165619s) [upload=523077]: Upload finished. err: <nil> Duration: 29.453941058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X........X................X.................X............] duration: 39.100305902s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:14:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 39.100330077s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X..] duration: 3.72068416s trace: https://jaeger.dev.storj.io/trace/7c51c2b650c865a4 FAIL (2025-12-30 12:14:05 UTC) orbiter warning: context deadline exceeded Duration: 3.720716521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XX.X.X.X........X...X..X..X.X....X.X.......XX......XX.XX.] duration: 5.999041336s trace: https://jaeger.dev.storj.io/trace/3773befad05e210c FAIL (2025-12-30 12:13:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 12:12:32.084099774 (596h43m31.955182819s) [upload=523077]: Starting upload Dec 30 12:12:32.084184265 (596h43m31.955267311s) [upload=523077]: Done starting upload Dec 30 12:12:32.084298102 (596h43m31.955381157s) [upload=523077]: Uploading segments... Dec 30 12:12:32.084460662 (596h43m31.955543697s) [upload=523077]: Got next segment. Inline: false Dec 30 12:12:32.084501960 (596h43m31.955585005s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:12:32.084507891 (596h43m31.955590936s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:12:32.329735714 (596h43m32.200818759s) [seg_pos={0 0}, upload=523077]: Begin upload segment... Dec 30 12:12:32.329821857 (596h43m32.200904903s) [seg_pos={0 0}, upload=523077]: Done begin upload segment. Dec 30 12:12:32.329867044 (596h43m32.200950099s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.329876352 (596h43m32.200959397s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330015867 (596h43m32.201098912s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330722923 (596h43m32.201805968s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330740678 (596h43m32.201823713s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331188230 (596h43m32.202271275s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331547804 (596h43m32.202630839s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331658455 (596h43m32.2027415s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332112069 (596h43m32.203195114s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332129633 (596h43m32.203212678s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332567826 (596h43m32.203650871s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332952449 (596h43m32.204035494s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333361859 (596h43m32.204444894s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333574864 (596h43m32.204657919s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334195897 (596h43m32.205278932s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334819795 (596h43m32.20590284s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:34.036767632 (596h43m33.907850677s) [upload=523077]: Next returned nil segment Dec 30 12:12:34.036801066 (596h43m33.907884101s) [upload=523077]: Waiting for error group managing segments... Dec 30 12:12:34.320104540 (596h43m34.191187575s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322221491 (596h43m34.193304526s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322663061 (596h43m34.193746106s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.329138792 (596h43m34.200221827s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.333089725 (596h43m34.20417277s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.336954995 (596h43m34.20803804s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.396856828 (596h43m34.267939873s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.403783257 (596h43m34.274866302s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.406970076 (596h43m34.278053121s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.412949751 (596h43m34.284032786s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.489634171 (596h43m34.360717206s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.669091115 (596h43m34.54017415s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681055958 (596h43m34.552139003s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681077820 (596h43m34.552160855s) [seg_pos={0 0}, upload=523077]: Segment reached optimal threshold of 13 pieces. Dec 30 12:12:34.681267250 (596h43m34.552350295s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection Dec 30 12:12:34.681277270 (596h43m34.552360325s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681532305 (596h43m34.55261534s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681548355 (596h43m34.55263139s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681568033 (596h43m34.552651068s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681575217 (596h43m34.552658252s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681591338 (596h43m34.552674383s) [seg_pos={0 0}, upload=523077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:12:34.681756532 (596h43m34.552839607s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:12:34.681764688 (596h43m34.552847733s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:12:34.983065131 (596h43m34.854148166s) [upload=523077]: Done uploading segments... Dec 30 12:12:34.983082584 (596h43m34.854165619s) [upload=523077]: Upload finished. err: <nil> Duration: 5.999082002s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X........X................X.................X...........] duration: 38.693140982s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:13:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 38.693169235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XX.X.X.X........X...X..X..X.X....X.X.......XX......XX.XX] duration: 30.00365615s trace: https://jaeger.dev.storj.io/trace/3773befad05e210c FAIL (2025-12-30 12:13:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 12:12:32.084099774 (596h43m31.955182819s) [upload=523077]: Starting upload Dec 30 12:12:32.084184265 (596h43m31.955267311s) [upload=523077]: Done starting upload Dec 30 12:12:32.084298102 (596h43m31.955381157s) [upload=523077]: Uploading segments... Dec 30 12:12:32.084460662 (596h43m31.955543697s) [upload=523077]: Got next segment. Inline: false Dec 30 12:12:32.084501960 (596h43m31.955585005s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:12:32.084507891 (596h43m31.955590936s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:12:32.329735714 (596h43m32.200818759s) [seg_pos={0 0}, upload=523077]: Begin upload segment... Dec 30 12:12:32.329821857 (596h43m32.200904903s) [seg_pos={0 0}, upload=523077]: Done begin upload segment. Dec 30 12:12:32.329867044 (596h43m32.200950099s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.329876352 (596h43m32.200959397s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330015867 (596h43m32.201098912s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330722923 (596h43m32.201805968s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.330740678 (596h43m32.201823713s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331188230 (596h43m32.202271275s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331547804 (596h43m32.202630839s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.331658455 (596h43m32.2027415s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332112069 (596h43m32.203195114s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332129633 (596h43m32.203212678s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332567826 (596h43m32.203650871s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.332952449 (596h43m32.204035494s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333361859 (596h43m32.204444894s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.333574864 (596h43m32.204657919s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334195897 (596h43m32.205278932s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:32.334819795 (596h43m32.20590284s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Uploading piece... Dec 30 12:12:34.036767632 (596h43m33.907850677s) [upload=523077]: Next returned nil segment Dec 30 12:12:34.036801066 (596h43m33.907884101s) [upload=523077]: Waiting for error group managing segments... Dec 30 12:12:34.320104540 (596h43m34.191187575s) [noise=true , address=145.79.227.13:4802 , piece_id=4UCTTQTRT7VMGDKEO2Q2FNP3WPFVL2GMZDX7JX67USROBN3H4IDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322221491 (596h43m34.193304526s) [noise=true , address=145.79.227.41:4085 , piece_id=ZE5IDDB5GMSSBX6PPAMNUEFCHLM5BGWMZEITUDJPLQTCVBLSLKYQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.322663061 (596h43m34.193746106s) [noise=true , address=145.79.227.21:4475 , piece_id=MPTGMVTSR5SVB4QRRHZLHCREKOKRBZYADKYRMURXEVIJA3VIOPNQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.329138792 (596h43m34.200221827s) [noise=true , address=145.79.227.36:4519 , piece_id=NV6XJ6FDUXVUVZNSA5YBL5CZ7TOKRKZMQ4VDQWG4PCQURLHNQEOQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.333089725 (596h43m34.20417277s) [noise=true , address=145.79.227.16:4911 , piece_id=JMC37HEWSWSYRM5R7RYG37YKSHYHMT2W2L72QSLWFEB5UUMJVEUA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.336954995 (596h43m34.20803804s) [noise=true , address=145.79.227.5:4223 , piece_id=Z6SPJJV6DF2GXWHYUWBN7GY722IGHWJBQCOE4JFUROIJMILDNT2Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.396856828 (596h43m34.267939873s) [noise=true , address=145.79.227.3:4149 , piece_id=BRUBBJXX7EECZXBHEAJFRH23RJ2OOM55WX3BIUUSRDNQGPRZQQ6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.403783257 (596h43m34.274866302s) [noise=true , address=145.79.227.25:4619 , piece_id=P3O6I6R663XCVPURX3KCTA4WNS7SESFZ4HOCHND7YRIQNWHQHW6A, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.406970076 (596h43m34.278053121s) [noise=true , address=145.79.227.51:4411 , piece_id=JGYLXNHXIBV4LYZWYHWTBJUHVGSC7H66NEEN6IAQKLIDQ6IRORFQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.412949751 (596h43m34.284032786s) [noise=true , address=145.79.227.47:4266 , piece_id=SQEUFQR7ERTSHNWL42ROX7S2B3E3CHY4CHIAUILITEUK4H6ESIYA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.489634171 (596h43m34.360717206s) [noise=true , address=77.90.31.13:4627 , piece_id=ZJCM6RLQOUHHYO5PYZXCHL2OFUG7GHYKTLSALGMDEQ4DUNGE4ZRQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.669091115 (596h43m34.54017415s) [noise=true , address=45.91.116.9:4261 , piece_id=ZG7T3YBICVASHE4WIUZP7VK4XAEP7HUC5BB6N2ZO45PUU4S4CJ6Q, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681055958 (596h43m34.552139003s) [noise=true , address=45.91.116.11:4119 , piece_id=HJVJRIZVWC6JJF45AXCEGQUKYMED53HS4WS3KIX2TCREHZCST2XA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: <nil> Dec 30 12:12:34.681077820 (596h43m34.552160855s) [seg_pos={0 0}, upload=523077]: Segment reached optimal threshold of 13 pieces. Dec 30 12:12:34.681267250 (596h43m34.552350295s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32892->45.91.116.12:4225: use of closed network connection Dec 30 12:12:34.681277270 (596h43m34.552360325s) [noise=true , address=45.91.116.12:4225 , piece_id=FKUBNQ23HV26DGTPPFCCQG25FZFTNOPCD3TJ55R3X2N2Y4NMZKJA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681532305 (596h43m34.55261534s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681548355 (596h43m34.55263139s) [noise=true , address=45.91.116.7:4009 , piece_id=E6UUUPBHJ33ECZZYX6U35MXDLOIYIDB65ZUAYTESQQTKFW2RPTEA, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681568033 (596h43m34.552651068s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 12:12:34.681575217 (596h43m34.552658252s) [noise=true , address=77.90.31.19:4844 , piece_id=JJDIVNEHXZVWAD22VEKZTFF7XH3WG4OL5HKRSYQKG7GQHGLOMJDQ, seg_pos={0 0}, upload=523077]: Long tail cancellation due to optimal threshold Dec 30 12:12:34.681591338 (596h43m34.552674383s) [seg_pos={0 0}, upload=523077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:12:34.681756532 (596h43m34.552839607s) [upload=523077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:12:34.681764688 (596h43m34.552847733s) [upload=523077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:12:34.983065131 (596h43m34.854148166s) [upload=523077]: Done uploading segments... Dec 30 12:12:34.983082584 (596h43m34.854165619s) [upload=523077]: Upload finished. err: <nil> Duration: 30.003679164s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X.] duration: 3.585899429s trace: https://jaeger.dev.storj.io/trace/7c51c2b650c865a4 FAIL (2025-12-30 12:12:53 UTC) orbiter warning: context deadline exceeded Duration: 3.585925007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.X........X................X.................X..........] duration: 56.395234819s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:12:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 56.395249626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....X] duration: 5.001211664s trace: https://jaeger.dev.storj.io/trace/7c51c2b650c865a4 FAIL (2025-12-30 12:12:25 UTC) orbiter warning: context deadline exceeded Duration: 5.001233946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....XX.X.X.X........X...X..X..X.X....X.X.......XX......XX.X] duration: 30.0037013s trace: https://jaeger.dev.storj.io/trace/6b73ce31b124f7a5 FAIL (2025-12-30 12:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: U3MW7UK4PL5NLWDDEO2BNOR2ACQHIICJBSTDKJFEQGFPQBK6H47A) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 6SRRWZZKREJX5OPAYWOIA2WQDEKU2IIWS4I7RD47BFQRCPVABRDQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: BUSX6WY6TEPFJCCUU7JSDRDQU577343GAYRPI4WY7Y4HK3RJA6IQ) context deadline exceeded ======LOGS====== ]: Starting upload Dec 30 12:11:19.623004378 (596h42m19.494087433s) [upload=523063]: Done starting upload Dec 30 12:11:19.623243263 (596h42m19.494326318s) [upload=523063]: Uploading segments... Dec 30 12:11:19.623356659 (596h42m19.494439694s) [upload=523063]: Got next segment. Inline: false Dec 30 12:11:19.623395322 (596h42m19.494478367s) [upload=523063]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:11:19.623402016 (596h42m19.494485071s) [upload=523063]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:11:19.868235617 (596h42m19.739318652s) [seg_pos={0 0}, upload=523063]: Begin upload segment... Dec 30 12:11:19.868305981 (596h42m19.739389026s) [seg_pos={0 0}, upload=523063]: Done begin upload segment. Dec 30 12:11:19.868312473 (596h42m19.739395508s) [noise=true , address=145.79.227.42:4123 , piece_id=6M4U2HP4OI63YLS5K243WSEPUUJXDH6UQ3IKANH4WZZADSDPFKPQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.868344514 (596h42m19.739427559s) [noise=true , address=145.79.227.41:4087 , piece_id=FIPQYMLA6VDUNYASNOTPYM2ABXLDQYC7UFCR2GV7YLQ6D7HDVK6Q, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.868834597 (596h42m19.739917632s) [noise=true , address=145.79.227.5:4223 , piece_id=F25FX65PYUD43AYIG5JDT7APOMICUKKZHBWAF4X2RNFDDELRB3PQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.869702831 (596h42m19.740785876s) [noise=true , address=45.91.116.11:4120 , piece_id=DLCVHK7BZ4WYART4FDNPNAQOPN45NR3UA3WJNUUTFQIDMLKUL5OQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.870682406 (596h42m19.741765451s) [noise=true , address=77.90.31.19:4844 , piece_id=6SRRWZZKREJX5OPAYWOIA2WQDEKU2IIWS4I7RD47BFQRCPVABRDQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.870951348 (596h42m19.742034393s) [noise=true , address=45.91.116.6:4974 , piece_id=U3MW7UK4PL5NLWDDEO2BNOR2ACQHIICJBSTDKJFEQGFPQBK6H47A, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.871100612 (596h42m19.742183647s) [noise=true , address=145.79.227.20:4437 , piece_id=6WZ5ZBFFHBM5FYYGUSC5MCSRQPVKVPNJL4R4CZSOCSTGLCZ67UIA, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.871542814 (596h42m19.742625859s) [noise=true , address=45.91.116.12:4226 , piece_id=BUSX6WY6TEPFJCCUU7JSDRDQU577343GAYRPI4WY7Y4HK3RJA6IQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.871761761 (596h42m19.742844806s) [noise=true , address=145.79.227.12:4765 , piece_id=DK3SGCD4SHIGVBYG55MXZZTOQDQQC75TYI544AFMY2HHPF5JMVLQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.872053507 (596h42m19.743136562s) [noise=true , address=145.79.227.50:4373 , piece_id=LQWOR44DCAPMQUAWRJO2ADJYSPJVVWIZLRD7C7ZZE65X6BU6OK5A, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.872500619 (596h42m19.743583654s) [noise=true , address=145.79.227.21:4475 , piece_id=7FHO33R3CPUSP5ZIZRJHOURBJ2IM2GAPHXW454MMWPXEZP7HXVYQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.872988858 (596h42m19.744071893s) [noise=true , address=145.79.227.49:4337 , piece_id=64DWCETTCXREKVE3JCN3DATWWLMH3YXZ2UAJKSY7IAIFHPXMOSZA, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.873188868 (596h42m19.744271914s) [noise=true , address=45.91.116.5:4299 , piece_id=H2S5QH7RV76Q3WZC75ZAOSQVGVBNQGHJZSRCIBUNNXYMGN7SOTQA, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.873721814 (596h42m19.744804859s) [noise=true , address=145.79.227.47:4267 , piece_id=VZC3MBDIXSG4PNNZBBQNEKMGXVWYZKH5O6CLO3LJTMNYC35M5UGA, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.874269727 (596h42m19.745352772s) [noise=true , address=145.79.227.17:4972 , piece_id=WOHIEFST4MA6YCA5NADETCQJQE7PUFZJQSG3ZPE5AS7NFXT5U7FQ, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:19.874460500 (596h42m19.745543535s) [noise=true , address=145.79.227.3:4150 , piece_id=ZMCANIJDMBJ2N2GXTOBFVHWWVCIOWCEUSCAB7X2JAFZCUECJMR3Q, seg_pos={0 0}, upload=523063]: Uploading piece... Dec 30 12:11:21.573995793 (596h42m21.445078838s) [upload=523063]: Next returned nil segment Dec 30 12:11:21.574013276 (596h42m21.445096321s) [upload=523063]: Waiting for error group managing segments... Dec 30 12:11:21.863634731 (596h42m21.734717776s) [noise=true , address=145.79.227.12:4765 , piece_id=DK3SGCD4SHIGVBYG55MXZZTOQDQQC75TYI544AFMY2HHPF5JMVLQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.864246045 (596h42m21.735329091s) [noise=true , address=145.79.227.50:4373 , piece_id=LQWOR44DCAPMQUAWRJO2ADJYSPJVVWIZLRD7C7ZZE65X6BU6OK5A, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.866264930 (596h42m21.737347975s) [noise=true , address=145.79.227.21:4475 , piece_id=7FHO33R3CPUSP5ZIZRJHOURBJ2IM2GAPHXW454MMWPXEZP7HXVYQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.866713603 (596h42m21.737796638s) [noise=true , address=145.79.227.20:4437 , piece_id=6WZ5ZBFFHBM5FYYGUSC5MCSRQPVKVPNJL4R4CZSOCSTGLCZ67UIA, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.866736467 (596h42m21.737819512s) [noise=true , address=145.79.227.47:4267 , piece_id=VZC3MBDIXSG4PNNZBBQNEKMGXVWYZKH5O6CLO3LJTMNYC35M5UGA, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.867367218 (596h42m21.738450253s) [noise=true , address=145.79.227.42:4123 , piece_id=6M4U2HP4OI63YLS5K243WSEPUUJXDH6UQ3IKANH4WZZADSDPFKPQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.883326547 (596h42m21.754409582s) [noise=true , address=145.79.227.41:4087 , piece_id=FIPQYMLA6VDUNYASNOTPYM2ABXLDQYC7UFCR2GV7YLQ6D7HDVK6Q, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.883959803 (596h42m21.755042838s) [noise=true , address=145.79.227.5:4223 , piece_id=F25FX65PYUD43AYIG5JDT7APOMICUKKZHBWAF4X2RNFDDELRB3PQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.942397537 (596h42m21.813480582s) [noise=true , address=145.79.227.3:4150 , piece_id=ZMCANIJDMBJ2N2GXTOBFVHWWVCIOWCEUSCAB7X2JAFZCUECJMR3Q, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:21.954529539 (596h42m21.825612584s) [noise=true , address=145.79.227.17:4972 , piece_id=WOHIEFST4MA6YCA5NADETCQJQE7PUFZJQSG3ZPE5AS7NFXT5U7FQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:22.048012075 (596h42m21.91909513s) [noise=true , address=77.90.31.19:4844 , piece_id=6SRRWZZKREJX5OPAYWOIA2WQDEKU2IIWS4I7RD47BFQRCPVABRDQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:22.173824922 (596h42m22.044907967s) [noise=true , address=45.91.116.12:4226 , piece_id=BUSX6WY6TEPFJCCUU7JSDRDQU577343GAYRPI4WY7Y4HK3RJA6IQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:22.199165705 (596h42m22.07024875s) [noise=true , address=45.91.116.6:4974 , piece_id=U3MW7UK4PL5NLWDDEO2BNOR2ACQHIICJBSTDKJFEQGFPQBK6H47A, seg_pos={0 0}, upload=523063]: Done uploading piece. err: <nil> Dec 30 12:11:22.199185723 (596h42m22.070268768s) [seg_pos={0 0}, upload=523063]: Segment reached optimal threshold of 13 pieces. Dec 30 12:11:22.199347832 (596h42m22.070430867s) [noise=true , address=45.91.116.5:4299 , piece_id=H2S5QH7RV76Q3WZC75ZAOSQVGVBNQGHJZSRCIBUNNXYMGN7SOTQA, seg_pos={0 0}, upload=523063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:43390->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:43390->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43390->45.91.116.5:4299: use of closed network connection Dec 30 12:11:22.199354344 (596h42m22.070437379s) [noise=true , address=45.91.116.5:4299 , piece_id=H2S5QH7RV76Q3WZC75ZAOSQVGVBNQGHJZSRCIBUNNXYMGN7SOTQA, seg_pos={0 0}, upload=523063]: Long tail cancellation due to optimal threshold Dec 30 12:11:22.199393749 (596h42m22.070476784s) [noise=true , address=145.79.227.49:4337 , piece_id=64DWCETTCXREKVE3JCN3DATWWLMH3YXZ2UAJKSY7IAIFHPXMOSZA, seg_pos={0 0}, upload=523063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 30 12:11:22.199402555 (596h42m22.07048559s) [noise=true , address=145.79.227.49:4337 , piece_id=64DWCETTCXREKVE3JCN3DATWWLMH3YXZ2UAJKSY7IAIFHPXMOSZA, seg_pos={0 0}, upload=523063]: Long tail cancellation due to optimal threshold Dec 30 12:11:22.199580625 (596h42m22.07066366s) [noise=true , address=45.91.116.11:4120 , piece_id=DLCVHK7BZ4WYART4FDNPNAQOPN45NR3UA3WJNUUTFQIDMLKUL5OQ, seg_pos={0 0}, upload=523063]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:53024->45.91.116.11:4120: use of closed network connection; context canceled Dec 30 12:11:22.199587538 (596h42m22.070670573s) [noise=true , address=45.91.116.11:4120 , piece_id=DLCVHK7BZ4WYART4FDNPNAQOPN45NR3UA3WJNUUTFQIDMLKUL5OQ, seg_pos={0 0}, upload=523063]: Long tail cancellation due to optimal threshold Dec 30 12:11:22.199602877 (596h42m22.070685912s) [seg_pos={0 0}, upload=523063]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:11:22.199668783 (596h42m22.070751818s) [upload=523063]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:11:22.199673742 (596h42m22.070756787s) [upload=523063]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:11:22.486666723 (596h42m22.357749758s) [upload=523063]: Done uploading segments... Dec 30 12:11:22.486687102 (596h42m22.357770137s) [upload=523063]: Upload finished. err: <nil> Duration: 30.00372168s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.....] duration: 3.839612075s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:11:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.839643746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.X........X................X.................X.........] duration: 42.112895504s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:11:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 42.112913839s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [....X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X....] duration: 3.846516639s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:10:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.846540526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X....X.X........X................X.................X........] duration: 38.613171254s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:10:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 38.613188067s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.....X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X...] duration: 3.875546672s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:10:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.875599421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX....XX.X.X.X........X...X..X..X.X....X.X.......XX......XX.] duration: 5.783742176s trace: https://jaeger.dev.storj.io/trace/492a9e8d5fc7dc42 FAIL (2025-12-30 12:10:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : false Dec 30 12:08:57.212307383 (596h39m57.083390418s) [upload=523027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:08:57.212311741 (596h39m57.083394776s) [upload=523027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:08:57.459667234 (596h39m57.330750269s) [seg_pos={0 0}, upload=523027]: Begin upload segment... Dec 30 12:08:57.459715666 (596h39m57.330798712s) [seg_pos={0 0}, upload=523027]: Done begin upload segment. Dec 30 12:08:57.459748138 (596h39m57.330831183s) [noise=true , address=145.79.227.52:4447 , piece_id=GYYS53W3I5AWFS4RZYKXJAJLYZOYNN7EIIFRE7EZ7RULGM4F2RNA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.459807671 (596h39m57.330890716s) [noise=true , address=145.79.227.47:4265 , piece_id=P7ANZHYVEGUK4XPD6YOPUU3Z6O3QE452QVK7XDFNMIPTBRF7ANNA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.460413305 (596h39m57.33149635s) [noise=true , address=145.79.227.10:4403 , piece_id=HGPYJZ5HLYREYHURJTCBPYAW7JAA3BV3PFK2L4SFZFY2NXLE74DQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.460922604 (596h39m57.332005639s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.461380026 (596h39m57.332463071s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462073897 (596h39m57.333156932s) [noise=true , address=77.90.31.21:4880 , piece_id=DRQJKEZZWZ5RLKK6IHI2H36ZQ25OJZFCNNHXPFGURVURS4IUTSZA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462176463 (596h39m57.333259498s) [noise=true , address=145.79.227.20:4437 , piece_id=AQBFBLAKB57YLPTDOBU4Q35OV3JO4DV77BMLOLBPHIABGKLEFCLA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462578488 (596h39m57.333661523s) [noise=true , address=45.91.116.12:4226 , piece_id=M2NGELGJGXIDYKI34WPG7UTDBGXSWK2533PN6IMEAAYF7S223XDA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462788808 (596h39m57.333871853s) [noise=true , address=45.91.116.7:4009 , piece_id=RY3RS3MIH6SB7IIKAB2EHZ535UIEMA7V4G5ZGGYLHK5YFUYW6TLA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462911692 (596h39m57.333994728s) [noise=true , address=145.79.227.11:4729 , piece_id=2QANA7EBWZ373DJ4UI6WDAEBKU7JFYVRYD2NRD6C6QEVUVE5UCCA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463286325 (596h39m57.33436937s) [noise=true , address=145.79.227.13:4803 , piece_id=2S3MUBR6BQJEGHHS7DUR2W3LGZ6EPVDFNBGUJYY2KW6QW2VEXHGQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463603229 (596h39m57.334686274s) [noise=true , address=45.91.116.5:4298 , piece_id=R4XKYCY366ROEMTQCUKPZZBW2DFWLRKYGPRRCAL5PRHR7NYEZQKA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463726333 (596h39m57.334809368s) [noise=true , address=145.79.227.7:4293 , piece_id=4YK3OS353A3UFCWTJE7DDOO3CEG5BUEC4CUUTEAJIQGHNSV3RPRA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464116827 (596h39m57.335199862s) [noise=true , address=145.79.227.45:4230 , piece_id=QX3NG7MHOZXWMWVXUKMHTG3ECG7HKVAKPW772OYD3V5NJ7XYU2IA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464391289 (596h39m57.335474334s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464453488 (596h39m57.335536533s) [noise=true , address=45.91.116.9:4264 , piece_id=5NPJU7J7BLWIMUFYN6ZXS73EZP6GL6RLERF6VK37L3542IBZALLQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:59.155709797 (596h39m59.026792842s) [upload=523027]: Next returned nil segment Dec 30 12:08:59.155730415 (596h39m59.02681346s) [upload=523027]: Waiting for error group managing segments... Dec 30 12:08:59.436845183 (596h39m59.307928218s) [noise=true , address=145.79.227.10:4403 , piece_id=HGPYJZ5HLYREYHURJTCBPYAW7JAA3BV3PFK2L4SFZFY2NXLE74DQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.439240203 (596h39m59.310323248s) [noise=true , address=145.79.227.45:4230 , piece_id=QX3NG7MHOZXWMWVXUKMHTG3ECG7HKVAKPW772OYD3V5NJ7XYU2IA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.443861913 (596h39m59.314944948s) [noise=true , address=145.79.227.11:4729 , piece_id=2QANA7EBWZ373DJ4UI6WDAEBKU7JFYVRYD2NRD6C6QEVUVE5UCCA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.451809968 (596h39m59.322893023s) [noise=true , address=145.79.227.20:4437 , piece_id=AQBFBLAKB57YLPTDOBU4Q35OV3JO4DV77BMLOLBPHIABGKLEFCLA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.467587110 (596h39m59.338670155s) [noise=true , address=145.79.227.13:4803 , piece_id=2S3MUBR6BQJEGHHS7DUR2W3LGZ6EPVDFNBGUJYY2KW6QW2VEXHGQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.480943984 (596h39m59.352027029s) [noise=true , address=145.79.227.47:4265 , piece_id=P7ANZHYVEGUK4XPD6YOPUU3Z6O3QE452QVK7XDFNMIPTBRF7ANNA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.537074725 (596h39m59.40815777s) [noise=true , address=145.79.227.52:4447 , piece_id=GYYS53W3I5AWFS4RZYKXJAJLYZOYNN7EIIFRE7EZ7RULGM4F2RNA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.770500565 (596h39m59.6415836s) [noise=true , address=145.79.227.7:4293 , piece_id=4YK3OS353A3UFCWTJE7DDOO3CEG5BUEC4CUUTEAJIQGHNSV3RPRA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.857879809 (596h39m59.728962844s) [noise=true , address=45.91.116.9:4264 , piece_id=5NPJU7J7BLWIMUFYN6ZXS73EZP6GL6RLERF6VK37L3542IBZALLQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.903950204 (596h39m59.775033249s) [noise=true , address=77.90.31.21:4880 , piece_id=DRQJKEZZWZ5RLKK6IHI2H36ZQ25OJZFCNNHXPFGURVURS4IUTSZA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.935948845 (596h39m59.80703189s) [noise=true , address=45.91.116.12:4226 , piece_id=M2NGELGJGXIDYKI34WPG7UTDBGXSWK2533PN6IMEAAYF7S223XDA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.955667292 (596h39m59.826750337s) [noise=true , address=45.91.116.5:4298 , piece_id=R4XKYCY366ROEMTQCUKPZZBW2DFWLRKYGPRRCAL5PRHR7NYEZQKA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:09:00.019740386 (596h39m59.890823421s) [noise=true , address=45.91.116.7:4009 , piece_id=RY3RS3MIH6SB7IIKAB2EHZ535UIEMA7V4G5ZGGYLHK5YFUYW6TLA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:09:00.019751929 (596h39m59.890834964s) [seg_pos={0 0}, upload=523027]: Segment reached optimal threshold of 13 pieces. Dec 30 12:09:00.019910320 (596h39m59.890993365s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection Dec 30 12:09:00.019918887 (596h39m59.891001922s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020351661 (596h39m59.891434706s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection Dec 30 12:09:00.020360327 (596h39m59.891443362s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020621204 (596h39m59.891704249s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:54168->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:09:00.020630592 (596h39m59.891713637s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020650039 (596h39m59.891733084s) [seg_pos={0 0}, upload=523027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:09:00.020713369 (596h39m59.891796414s) [upload=523027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:09:00.020718800 (596h39m59.891801835s) [upload=523027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:09:00.308291754 (596h40m0.179374789s) [upload=523027]: Done uploading segments... Dec 30 12:09:00.308310119 (596h40m0.179393164s) [upload=523027]: Upload finished. err: <nil> Duration: 5.783784056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X....X.X........X................X.................X.......] duration: 32.64362367s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:10:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 32.643655942s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X..] duration: 3.547215526s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:09:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547261785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX......XX] duration: 30.002808053s trace: https://jaeger.dev.storj.io/trace/492a9e8d5fc7dc42 FAIL (2025-12-30 12:09:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : false Dec 30 12:08:57.212307383 (596h39m57.083390418s) [upload=523027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:08:57.212311741 (596h39m57.083394776s) [upload=523027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:08:57.459667234 (596h39m57.330750269s) [seg_pos={0 0}, upload=523027]: Begin upload segment... Dec 30 12:08:57.459715666 (596h39m57.330798712s) [seg_pos={0 0}, upload=523027]: Done begin upload segment. Dec 30 12:08:57.459748138 (596h39m57.330831183s) [noise=true , address=145.79.227.52:4447 , piece_id=GYYS53W3I5AWFS4RZYKXJAJLYZOYNN7EIIFRE7EZ7RULGM4F2RNA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.459807671 (596h39m57.330890716s) [noise=true , address=145.79.227.47:4265 , piece_id=P7ANZHYVEGUK4XPD6YOPUU3Z6O3QE452QVK7XDFNMIPTBRF7ANNA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.460413305 (596h39m57.33149635s) [noise=true , address=145.79.227.10:4403 , piece_id=HGPYJZ5HLYREYHURJTCBPYAW7JAA3BV3PFK2L4SFZFY2NXLE74DQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.460922604 (596h39m57.332005639s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.461380026 (596h39m57.332463071s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462073897 (596h39m57.333156932s) [noise=true , address=77.90.31.21:4880 , piece_id=DRQJKEZZWZ5RLKK6IHI2H36ZQ25OJZFCNNHXPFGURVURS4IUTSZA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462176463 (596h39m57.333259498s) [noise=true , address=145.79.227.20:4437 , piece_id=AQBFBLAKB57YLPTDOBU4Q35OV3JO4DV77BMLOLBPHIABGKLEFCLA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462578488 (596h39m57.333661523s) [noise=true , address=45.91.116.12:4226 , piece_id=M2NGELGJGXIDYKI34WPG7UTDBGXSWK2533PN6IMEAAYF7S223XDA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462788808 (596h39m57.333871853s) [noise=true , address=45.91.116.7:4009 , piece_id=RY3RS3MIH6SB7IIKAB2EHZ535UIEMA7V4G5ZGGYLHK5YFUYW6TLA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.462911692 (596h39m57.333994728s) [noise=true , address=145.79.227.11:4729 , piece_id=2QANA7EBWZ373DJ4UI6WDAEBKU7JFYVRYD2NRD6C6QEVUVE5UCCA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463286325 (596h39m57.33436937s) [noise=true , address=145.79.227.13:4803 , piece_id=2S3MUBR6BQJEGHHS7DUR2W3LGZ6EPVDFNBGUJYY2KW6QW2VEXHGQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463603229 (596h39m57.334686274s) [noise=true , address=45.91.116.5:4298 , piece_id=R4XKYCY366ROEMTQCUKPZZBW2DFWLRKYGPRRCAL5PRHR7NYEZQKA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.463726333 (596h39m57.334809368s) [noise=true , address=145.79.227.7:4293 , piece_id=4YK3OS353A3UFCWTJE7DDOO3CEG5BUEC4CUUTEAJIQGHNSV3RPRA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464116827 (596h39m57.335199862s) [noise=true , address=145.79.227.45:4230 , piece_id=QX3NG7MHOZXWMWVXUKMHTG3ECG7HKVAKPW772OYD3V5NJ7XYU2IA, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464391289 (596h39m57.335474334s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:57.464453488 (596h39m57.335536533s) [noise=true , address=45.91.116.9:4264 , piece_id=5NPJU7J7BLWIMUFYN6ZXS73EZP6GL6RLERF6VK37L3542IBZALLQ, seg_pos={0 0}, upload=523027]: Uploading piece... Dec 30 12:08:59.155709797 (596h39m59.026792842s) [upload=523027]: Next returned nil segment Dec 30 12:08:59.155730415 (596h39m59.02681346s) [upload=523027]: Waiting for error group managing segments... Dec 30 12:08:59.436845183 (596h39m59.307928218s) [noise=true , address=145.79.227.10:4403 , piece_id=HGPYJZ5HLYREYHURJTCBPYAW7JAA3BV3PFK2L4SFZFY2NXLE74DQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.439240203 (596h39m59.310323248s) [noise=true , address=145.79.227.45:4230 , piece_id=QX3NG7MHOZXWMWVXUKMHTG3ECG7HKVAKPW772OYD3V5NJ7XYU2IA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.443861913 (596h39m59.314944948s) [noise=true , address=145.79.227.11:4729 , piece_id=2QANA7EBWZ373DJ4UI6WDAEBKU7JFYVRYD2NRD6C6QEVUVE5UCCA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.451809968 (596h39m59.322893023s) [noise=true , address=145.79.227.20:4437 , piece_id=AQBFBLAKB57YLPTDOBU4Q35OV3JO4DV77BMLOLBPHIABGKLEFCLA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.467587110 (596h39m59.338670155s) [noise=true , address=145.79.227.13:4803 , piece_id=2S3MUBR6BQJEGHHS7DUR2W3LGZ6EPVDFNBGUJYY2KW6QW2VEXHGQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.480943984 (596h39m59.352027029s) [noise=true , address=145.79.227.47:4265 , piece_id=P7ANZHYVEGUK4XPD6YOPUU3Z6O3QE452QVK7XDFNMIPTBRF7ANNA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.537074725 (596h39m59.40815777s) [noise=true , address=145.79.227.52:4447 , piece_id=GYYS53W3I5AWFS4RZYKXJAJLYZOYNN7EIIFRE7EZ7RULGM4F2RNA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.770500565 (596h39m59.6415836s) [noise=true , address=145.79.227.7:4293 , piece_id=4YK3OS353A3UFCWTJE7DDOO3CEG5BUEC4CUUTEAJIQGHNSV3RPRA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.857879809 (596h39m59.728962844s) [noise=true , address=45.91.116.9:4264 , piece_id=5NPJU7J7BLWIMUFYN6ZXS73EZP6GL6RLERF6VK37L3542IBZALLQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.903950204 (596h39m59.775033249s) [noise=true , address=77.90.31.21:4880 , piece_id=DRQJKEZZWZ5RLKK6IHI2H36ZQ25OJZFCNNHXPFGURVURS4IUTSZA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.935948845 (596h39m59.80703189s) [noise=true , address=45.91.116.12:4226 , piece_id=M2NGELGJGXIDYKI34WPG7UTDBGXSWK2533PN6IMEAAYF7S223XDA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:08:59.955667292 (596h39m59.826750337s) [noise=true , address=45.91.116.5:4298 , piece_id=R4XKYCY366ROEMTQCUKPZZBW2DFWLRKYGPRRCAL5PRHR7NYEZQKA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:09:00.019740386 (596h39m59.890823421s) [noise=true , address=45.91.116.7:4009 , piece_id=RY3RS3MIH6SB7IIKAB2EHZ535UIEMA7V4G5ZGGYLHK5YFUYW6TLA, seg_pos={0 0}, upload=523027]: Done uploading piece. err: <nil> Dec 30 12:09:00.019751929 (596h39m59.890834964s) [seg_pos={0 0}, upload=523027]: Segment reached optimal threshold of 13 pieces. Dec 30 12:09:00.019910320 (596h39m59.890993365s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58948->45.91.116.8:4045: use of closed network connection Dec 30 12:09:00.019918887 (596h39m59.891001922s) [noise=true , address=45.91.116.8:4045 , piece_id=MUSBC35N4CP3MH3KJA25RMTN2FLJ4JLSYGYDR27UJQ2WPON4A7EQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020351661 (596h39m59.891434706s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45340->45.91.116.11:4118: use of closed network connection Dec 30 12:09:00.020360327 (596h39m59.891443362s) [noise=true , address=45.91.116.11:4118 , piece_id=TRF5FVW7TDPD5BZOCLTWVI74MPSRIXEUYF6TYCNCCJ3WWTGFCEBQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020621204 (596h39m59.891704249s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:54168->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 12:09:00.020630592 (596h39m59.891713637s) [noise=true , address=45.91.116.14:4189 , piece_id=P2HLEEIX46272KX2HL6EGWI2TZVGSELFTKMVQG2KS5RB4IN7HZKQ, seg_pos={0 0}, upload=523027]: Long tail cancellation due to optimal threshold Dec 30 12:09:00.020650039 (596h39m59.891733084s) [seg_pos={0 0}, upload=523027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:09:00.020713369 (596h39m59.891796414s) [upload=523027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:09:00.020718800 (596h39m59.891801835s) [upload=523027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:09:00.308291754 (596h40m0.179374789s) [upload=523027]: Done uploading segments... Dec 30 12:09:00.308310119 (596h40m0.179393164s) [upload=523027]: Upload finished. err: <nil> Duration: 30.002830275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X....X.X........X................X.................X......] duration: 26.393317696s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:09:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 26.393347854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X.] duration: 3.725351224s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:08:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.725374418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X....X.X........X................X.................X.....] duration: 53.405820933s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:08:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 53.405841862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...X] duration: 5.000848509s trace: https://jaeger.dev.storj.io/trace/1f659650b5478edb FAIL (2025-12-30 12:08:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000873246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX......X] duration: 30.00246224s trace: https://jaeger.dev.storj.io/trace/5d7bcf9f3c5a1a64 FAIL (2025-12-30 12:08:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: YTLXHLPTT2KHVBP7U7FXXVFQ3BKUKC64O6GZRIDNMGOBWNY7E2FA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: R243LP647T6KL5LJZDTAMTRIEFLOCW2ZOVH6KA5FEH7YFE6HK5EQ) context deadline exceeded ======LOGS====== Dec 30 12:08:03.131314867 (596h39m3.002397912s) [upload=523017]: Starting upload Dec 30 12:08:03.131419396 (596h39m3.002502441s) [upload=523017]: Done starting upload Dec 30 12:08:03.131513365 (596h39m3.00259641s) [upload=523017]: Uploading segments... Dec 30 12:08:03.131661838 (596h39m3.002744883s) [upload=523017]: Got next segment. Inline: false Dec 30 12:08:03.131702776 (596h39m3.002785831s) [upload=523017]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:08:03.131709298 (596h39m3.002792343s) [upload=523017]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:08:03.387728950 (596h39m3.258811985s) [seg_pos={0 0}, upload=523017]: Begin upload segment... Dec 30 12:08:03.387789816 (596h39m3.258872861s) [seg_pos={0 0}, upload=523017]: Done begin upload segment. Dec 30 12:08:03.387833669 (596h39m3.258916714s) [noise=true , address=145.79.227.35:4483 , piece_id=FNSBVUCMAVRTIDBJRXG3WAI4VGFBOMSH4RFVFWTDLCLO2KYPAIGA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.387851844 (596h39m3.258934889s) [noise=true , address=77.90.31.19:4842 , piece_id=YTLXHLPTT2KHVBP7U7FXXVFQ3BKUKC64O6GZRIDNMGOBWNY7E2FA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.388429764 (596h39m3.259512809s) [noise=true , address=145.79.227.52:4447 , piece_id=SRICKXI72QN2YMKVWB2ROKYD63BAZ5L2SUOLADIXMM3ZW7Z2MDVA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.388789320 (596h39m3.259872365s) [noise=true , address=77.90.31.10:4519 , piece_id=IPHI4CLPOAZ5BCDBPBKLLVQAUB65VFIFKLAKE7RMH6GF3ERSTA7A, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.389013396 (596h39m3.260096441s) [noise=true , address=145.79.227.17:4971 , piece_id=6KSLAVH2E33RULN67EOKYZUWMYXYNQTIBOUSJA6CDRMAYA2E4LWA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.389194500 (596h39m3.260277545s) [noise=true , address=145.79.227.16:4912 , piece_id=NX47O5R5HJ5BEQ4S6NUAG23AIYVP2R64NYCT4QB7RGWLKC7TDSVA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.389583601 (596h39m3.260666636s) [noise=true , address=45.91.116.10:4083 , piece_id=R243LP647T6KL5LJZDTAMTRIEFLOCW2ZOVH6KA5FEH7YFE6HK5EQ, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.389622286 (596h39m3.260705331s) [noise=true , address=77.90.31.21:4878 , piece_id=I6ZYJXHDZAKEALHXFLCTWZXEKQ77O6LLY75JDZS6VUT2RHBR3AQQ, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390119342 (596h39m3.261202377s) [noise=true , address=145.79.227.21:4475 , piece_id=LXYVYX67A7U7ZQV2KT3RQTSWCOSA6J35E6YG52S4VTOKQM3POYXA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390334000 (596h39m3.261417036s) [noise=true , address=145.79.227.39:4013 , piece_id=VR4YGLG6Y2LHUC7Q3WZJV7MZ564NX3UCFFRMIYU742ACUC4SCBWA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390432168 (596h39m3.261515203s) [noise=true , address=145.79.227.6:4258 , piece_id=ONHJ7P45GK2NCP4JFRB2VOEGBO3JUQUSUBCEL3MPDN6NAVDB2N3Q, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390766844 (596h39m3.261849879s) [noise=true , address=145.79.227.10:4403 , piece_id=ABTDMPJ3GHABTVAAL4WW36L2M4I5SROZODYFG5YMLCJXHFC3ZMRQ, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.390872896 (596h39m3.261955941s) [noise=true , address=145.79.227.23:4545 , piece_id=QKHYQYE332CEBVFZA4W7RKRDJ6JDWPLTAANIYZOCGDQJ3JOIT3HA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.391074681 (596h39m3.262157726s) [noise=true , address=145.79.227.36:4518 , piece_id=QUYOJR6VWPXPKMYZMTEXHF7K6QO2XYYFVHXKEDNIGBMZRRU3OBKQ, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.391429867 (596h39m3.262512912s) [noise=true , address=145.79.227.51:4409 , piece_id=ZCBHG3WRIZ5K243KRZZL4SEFV7LU4WJC32UOK3PXAI7XVNBZAQVA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:03.391983150 (596h39m3.263066195s) [noise=true , address=45.91.116.6:4974 , piece_id=V4LTRBNVA4ZOQNLIRVWTV3A7A3DEAMZIHZVXXM3XVFP4D7ACIEIA, seg_pos={0 0}, upload=523017]: Uploading piece... Dec 30 12:08:05.095873405 (596h39m4.96695644s) [upload=523017]: Next returned nil segment Dec 30 12:08:05.095887863 (596h39m4.966970899s) [upload=523017]: Waiting for error group managing segments... Dec 30 12:08:05.380624156 (596h39m5.251707191s) [noise=true , address=145.79.227.6:4258 , piece_id=ONHJ7P45GK2NCP4JFRB2VOEGBO3JUQUSUBCEL3MPDN6NAVDB2N3Q, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.382375320 (596h39m5.253458355s) [noise=true , address=145.79.227.39:4013 , piece_id=VR4YGLG6Y2LHUC7Q3WZJV7MZ564NX3UCFFRMIYU742ACUC4SCBWA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.391486329 (596h39m5.262569364s) [noise=true , address=145.79.227.52:4447 , piece_id=SRICKXI72QN2YMKVWB2ROKYD63BAZ5L2SUOLADIXMM3ZW7Z2MDVA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.394502492 (596h39m5.265585537s) [noise=true , address=145.79.227.36:4518 , piece_id=QUYOJR6VWPXPKMYZMTEXHF7K6QO2XYYFVHXKEDNIGBMZRRU3OBKQ, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.398363314 (596h39m5.269446349s) [noise=true , address=145.79.227.17:4971 , piece_id=6KSLAVH2E33RULN67EOKYZUWMYXYNQTIBOUSJA6CDRMAYA2E4LWA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.400584834 (596h39m5.271667879s) [noise=true , address=145.79.227.10:4403 , piece_id=ABTDMPJ3GHABTVAAL4WW36L2M4I5SROZODYFG5YMLCJXHFC3ZMRQ, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.401015754 (596h39m5.272098799s) [noise=true , address=145.79.227.51:4409 , piece_id=ZCBHG3WRIZ5K243KRZZL4SEFV7LU4WJC32UOK3PXAI7XVNBZAQVA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.406963680 (596h39m5.278046725s) [noise=true , address=145.79.227.23:4545 , piece_id=QKHYQYE332CEBVFZA4W7RKRDJ6JDWPLTAANIYZOCGDQJ3JOIT3HA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.408678004 (596h39m5.27976105s) [noise=true , address=145.79.227.21:4475 , piece_id=LXYVYX67A7U7ZQV2KT3RQTSWCOSA6J35E6YG52S4VTOKQM3POYXA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.423604858 (596h39m5.294687903s) [noise=true , address=145.79.227.16:4912 , piece_id=NX47O5R5HJ5BEQ4S6NUAG23AIYVP2R64NYCT4QB7RGWLKC7TDSVA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.552469587 (596h39m5.423552632s) [noise=true , address=77.90.31.19:4842 , piece_id=YTLXHLPTT2KHVBP7U7FXXVFQ3BKUKC64O6GZRIDNMGOBWNY7E2FA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.675134440 (596h39m5.546217485s) [noise=true , address=77.90.31.10:4519 , piece_id=IPHI4CLPOAZ5BCDBPBKLLVQAUB65VFIFKLAKE7RMH6GF3ERSTA7A, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.791038610 (596h39m5.662121655s) [noise=true , address=45.91.116.10:4083 , piece_id=R243LP647T6KL5LJZDTAMTRIEFLOCW2ZOVH6KA5FEH7YFE6HK5EQ, seg_pos={0 0}, upload=523017]: Done uploading piece. err: <nil> Dec 30 12:08:05.791058307 (596h39m5.662141342s) [seg_pos={0 0}, upload=523017]: Segment reached optimal threshold of 13 pieces. Dec 30 12:08:05.791238139 (596h39m5.662321184s) [noise=true , address=145.79.227.35:4483 , piece_id=FNSBVUCMAVRTIDBJRXG3WAI4VGFBOMSH4RFVFWTDLCLO2KYPAIGA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 12:08:05.791244412 (596h39m5.662327457s) [noise=true , address=145.79.227.35:4483 , piece_id=FNSBVUCMAVRTIDBJRXG3WAI4VGFBOMSH4RFVFWTDLCLO2KYPAIGA, seg_pos={0 0}, upload=523017]: Long tail cancellation due to optimal threshold Dec 30 12:08:05.791278928 (596h39m5.662361963s) [noise=true , address=45.91.116.6:4974 , piece_id=V4LTRBNVA4ZOQNLIRVWTV3A7A3DEAMZIHZVXXM3XVFP4D7ACIEIA, seg_pos={0 0}, upload=523017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:46000->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:46000->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46000->45.91.116.6:4974: use of closed network connection Dec 30 12:08:05.791287404 (596h39m5.662370439s) [noise=true , address=45.91.116.6:4974 , piece_id=V4LTRBNVA4ZOQNLIRVWTV3A7A3DEAMZIHZVXXM3XVFP4D7ACIEIA, seg_pos={0 0}, upload=523017]: Long tail cancellation due to optimal threshold Dec 30 12:08:05.791424244 (596h39m5.662507289s) [noise=true , address=77.90.31.21:4878 , piece_id=I6ZYJXHDZAKEALHXFLCTWZXEKQ77O6LLY75JDZS6VUT2RHBR3AQQ, seg_pos={0 0}, upload=523017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 30 12:08:05.791430436 (596h39m5.662513481s) [noise=true , address=77.90.31.21:4878 , piece_id=I6ZYJXHDZAKEALHXFLCTWZXEKQ77O6LLY75JDZS6VUT2RHBR3AQQ, seg_pos={0 0}, upload=523017]: Long tail cancellation due to optimal threshold Dec 30 12:08:05.791444963 (596h39m5.662528008s) [seg_pos={0 0}, upload=523017]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:08:05.791508534 (596h39m5.662591579s) [upload=523017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:08:05.791513323 (596h39m5.662596358s) [upload=523017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:08:06.086708351 (596h39m5.957791396s) [upload=523017]: Done uploading segments... Dec 30 12:08:06.086730914 (596h39m5.957813959s) [upload=523017]: Upload finished. err: <nil> Duration: 30.00248849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X....X.X........X................X.................X....] duration: 23.157201568s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 23.157221115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX......] duration: 5.736542561s trace: https://jaeger.dev.storj.io/trace/36fe60c4769b78 FAIL (2025-12-30 12:07:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ) context deadline exceeded ======LOGS====== Dec 30 12:02:53.402447340 (596h33m53.273530386s) [upload=522940]: Starting upload Dec 30 12:02:53.402520319 (596h33m53.273603354s) [upload=522940]: Done starting upload Dec 30 12:02:53.402623215 (596h33m53.27370626s) [upload=522940]: Uploading segments... Dec 30 12:02:53.402741190 (596h33m53.273824235s) [upload=522940]: Got next segment. Inline: false Dec 30 12:02:53.402778300 (596h33m53.273861335s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:02:53.402783891 (596h33m53.273866926s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:02:53.642416319 (596h33m53.513499354s) [seg_pos={0 0}, upload=522940]: Begin upload segment... Dec 30 12:02:53.642463549 (596h33m53.513546584s) [seg_pos={0 0}, upload=522940]: Done begin upload segment. Dec 30 12:02:53.642492083 (596h33m53.513575128s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642522181 (596h33m53.513605226s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642868831 (596h33m53.513951866s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643219980 (596h33m53.514303025s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643502698 (596h33m53.514585733s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643916997 (596h33m53.515000042s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644254249 (596h33m53.515337284s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644663448 (596h33m53.515746483s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.645423044 (596h33m53.516506089s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.646902191 (596h33m53.517985236s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.647439875 (596h33m53.51852291s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.648684635 (596h33m53.51976768s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.650246178 (596h33m53.521329213s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.651668687 (596h33m53.522751722s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.652945979 (596h33m53.524029024s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.654504297 (596h33m53.525587342s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:55.342521027 (596h33m55.213604062s) [upload=522940]: Next returned nil segment Dec 30 12:02:55.342538249 (596h33m55.213621284s) [upload=522940]: Waiting for error group managing segments... Dec 30 12:02:55.626493855 (596h33m55.4975769s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.638374107 (596h33m55.509457142s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.639801966 (596h33m55.510885011s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.643779369 (596h33m55.514862414s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.647465909 (596h33m55.518548954s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.652589877 (596h33m55.523672912s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.660725177 (596h33m55.531808212s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.665156515 (596h33m55.53623956s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.666087949 (596h33m55.537170994s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.681336385 (596h33m55.55241943s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.017694183 (596h33m55.888777228s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.060133734 (596h33m55.93121677s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858555196 (596h33m56.729638251s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858584001 (596h33m56.729667036s) [seg_pos={0 0}, upload=522940]: Segment reached optimal threshold of 13 pieces. Dec 30 12:02:56.858912295 (596h33m56.729995331s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858939878 (596h33m56.730022913s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.858977289 (596h33m56.730060334s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858995924 (596h33m56.730078959s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859161951 (596h33m56.730244986s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.859172521 (596h33m56.730255556s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859205423 (596h33m56.730288458s) [seg_pos={0 0}, upload=522940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:02:56.859313780 (596h33m56.730396815s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:02:56.859322136 (596h33m56.730405181s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:02:57.152351941 (596h33m57.023434976s) [upload=522940]: Done uploading segments... Dec 30 12:02:57.152371689 (596h33m57.023454734s) [upload=522940]: Upload finished. err: <nil> Duration: 5.736567578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X...] duration: 3.681825366s trace: https://jaeger.dev.storj.io/trace/7a477c4bbcbc4acd FAIL (2025-12-30 12:07:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.681854963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X....X.X........X................X.................X...] duration: 36.025568011s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:07:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 36.025585033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..XXX....XX.X.X.X........X...X..X..X.X....X.X.......XX.....] duration: 5.690081162s trace: https://jaeger.dev.storj.io/trace/36fe60c4769b78 FAIL (2025-12-30 12:07:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ) context deadline exceeded ======LOGS====== Dec 30 12:02:53.402447340 (596h33m53.273530386s) [upload=522940]: Starting upload Dec 30 12:02:53.402520319 (596h33m53.273603354s) [upload=522940]: Done starting upload Dec 30 12:02:53.402623215 (596h33m53.27370626s) [upload=522940]: Uploading segments... Dec 30 12:02:53.402741190 (596h33m53.273824235s) [upload=522940]: Got next segment. Inline: false Dec 30 12:02:53.402778300 (596h33m53.273861335s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 12:02:53.402783891 (596h33m53.273866926s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 12:02:53.642416319 (596h33m53.513499354s) [seg_pos={0 0}, upload=522940]: Begin upload segment... Dec 30 12:02:53.642463549 (596h33m53.513546584s) [seg_pos={0 0}, upload=522940]: Done begin upload segment. Dec 30 12:02:53.642492083 (596h33m53.513575128s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642522181 (596h33m53.513605226s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.642868831 (596h33m53.513951866s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643219980 (596h33m53.514303025s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643502698 (596h33m53.514585733s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.643916997 (596h33m53.515000042s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644254249 (596h33m53.515337284s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.644663448 (596h33m53.515746483s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.645423044 (596h33m53.516506089s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.646902191 (596h33m53.517985236s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.647439875 (596h33m53.51852291s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.648684635 (596h33m53.51976768s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.650246178 (596h33m53.521329213s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.651668687 (596h33m53.522751722s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.652945979 (596h33m53.524029024s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:53.654504297 (596h33m53.525587342s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Uploading piece... Dec 30 12:02:55.342521027 (596h33m55.213604062s) [upload=522940]: Next returned nil segment Dec 30 12:02:55.342538249 (596h33m55.213621284s) [upload=522940]: Waiting for error group managing segments... Dec 30 12:02:55.626493855 (596h33m55.4975769s) [noise=true , address=145.79.227.19:4077 , piece_id=TEJZL6BQSMVGRENA4VDMWIHNKNEHCY2UTIWUGLFKOVSTSITGWUZA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.638374107 (596h33m55.509457142s) [noise=true , address=145.79.227.20:4437 , piece_id=2WKH4PQIK5XCFQLVK3772GT5LZ4KRBL6SECDNBGJXXHJ2FAKAPIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.639801966 (596h33m55.510885011s) [noise=true , address=145.79.227.32:4834 , piece_id=HPMCPRY6ITYRZTCVQG7KRG6XKOC5BOA43NG3WBFRKVJXSPGQLEKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.643779369 (596h33m55.514862414s) [noise=true , address=145.79.227.50:4373 , piece_id=FI5W3NKZWYBIP3WR75HFIA3WQE7PTVXH2GACFPSJLUDQFK67GARA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.647465909 (596h33m55.518548954s) [noise=true , address=145.79.227.14:4839 , piece_id=4IEIUZBVMLNFVWW7SVZQGC5HITDR6XQFAZXU3XL3NS35HNJEJJIA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.652589877 (596h33m55.523672912s) [noise=true , address=145.79.227.2:4115 , piece_id=GYJJ42KONVDPOIQ3WIVBWE6W57J54QXU2VFYQDLL5HPXVZBDP5DA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.660725177 (596h33m55.531808212s) [noise=true , address=145.79.227.5:4222 , piece_id=2FCUSQGYEFV7GL2N3IFOWPHUKGUJ2FSRF64XRYWASMOXWRHHB3JQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.665156515 (596h33m55.53623956s) [noise=true , address=145.79.227.26:4653 , piece_id=3M6VTXTH6AE4LKG44GLI4A53K752E5FN6GSOWZ5KDI2536GQQMWA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.666087949 (596h33m55.537170994s) [noise=true , address=145.79.227.16:4912 , piece_id=ZGL2K7OW574HN5WQYAKPRS2BW3STZT77GCUA72DM2YAU77MSGAQA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:55.681336385 (596h33m55.55241943s) [noise=true , address=145.79.227.38:4979 , piece_id=2A6JTGELITNNFJBWHDICXTFM35NTGO2AIXWCRUTHHLSQBKVX4APQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.017694183 (596h33m55.888777228s) [noise=true , address=45.91.116.12:4227 , piece_id=6TFKYLC3ALRQQHZG3VTWBPVJLWNLCKC6CG5EAX4BDMN74S2ROWKA, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.060133734 (596h33m55.93121677s) [noise=true , address=45.91.116.11:4119 , piece_id=4NNNDAI7DBRAU7U2F4CTFGKMG73HB3K5J5JUW3DBHB2SPY5ARERQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858555196 (596h33m56.729638251s) [noise=true , address=45.91.116.10:4083 , piece_id=6QRUUZFTXXVDSMFMDU7SXUYKQ7B5U5ZO5DBXX4RBSVZEJWONE7YQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: <nil> Dec 30 12:02:56.858584001 (596h33m56.729667036s) [seg_pos={0 0}, upload=522940]: Segment reached optimal threshold of 13 pieces. Dec 30 12:02:56.858912295 (596h33m56.729995331s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858939878 (596h33m56.730022913s) [noise=true , address=45.91.116.6:4975 , piece_id=HPJVWV6XQZ2243VZICXC2BAOKM7GV3ZRSOIWYDJX3QUUIGY7JZIQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.858977289 (596h33m56.730060334s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.858995924 (596h33m56.730078959s) [noise=true , address=145.79.227.43:4157 , piece_id=3HVN5WJI6O5XH4LPYRW75YT6UVNUAND25DKXOHUMT5WJQJV37RTQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859161951 (596h33m56.730244986s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 12:02:56.859172521 (596h33m56.730255556s) [noise=true , address=145.79.227.40:4049 , piece_id=ULCSUROYPA3AJQSTBG7PZLQRUED2DONBUWPHSPLLP2ODCPWUTKQQ, seg_pos={0 0}, upload=522940]: Long tail cancellation due to optimal threshold Dec 30 12:02:56.859205423 (596h33m56.730288458s) [seg_pos={0 0}, upload=522940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 12:02:56.859313780 (596h33m56.730396815s) [upload=522940]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:02:56.859322136 (596h33m56.730405181s) [upload=522940]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:02:57.152351941 (596h33m57.023434976s) [upload=522940]: Done uploading segments... Dec 30 12:02:57.152371689 (596h33m57.023454734s) [upload=522940]: Upload finished. err: <nil> Duration: 5.690110666s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XX.......X..XX.X...X.X..X..X.X.X...X....XX.XXXXXXXX.X...X..] duration: 3.571350875s trace: https://jaeger.dev.storj.io/trace/7a477c4bbcbc4acd FAIL (2025-12-30 12:06:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571377836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X....X.X........X................X.................X..] duration: 40.515802939s trace: https://jaeger.dev.storj.io/trace/415656ec33c5053e FAIL (2025-12-30 12:06:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, Piece ID: EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA) context deadline exceeded; piecestore: (Node ID: 12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, Piece ID: FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA) context deadline exceeded; piecestore: (Node ID: 1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B, Piece ID: AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ) context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.751728830 (596h35m12.622811875s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752145363 (596h35m12.623228408s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752539955 (596h35m12.623623s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.752923716 (596h35m12.624006761s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753313738 (596h35m12.624396783s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.753703751 (596h35m12.624786796s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.754802814 (596h35m12.625885849s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755239445 (596h35m12.62632249s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.755749136 (596h35m12.626832171s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.756273163 (596h35m12.627356209s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.758050107 (596h35m12.629133142s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.759843753 (596h35m12.630926798s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.761593504 (596h35m12.63267655s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.763327867 (596h35m12.634410912s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:12.764983519 (596h35m12.636066564s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Uploading piece... Dec 30 12:04:14.216391518 (596h35m14.087474554s) [upload=522962]: Next returned nil segment Dec 30 12:04:14.216406577 (596h35m14.087489612s) [upload=522962]: Waiting for error group managing segments... Dec 30 12:04:14.618058225 (596h35m14.48914126s) [noise=true , address=145.79.227.43:4179 , piece_id=VGABZJZ27GQMGN3E4Q64WHWNG255DCOCHGIH2XHQFBSARVAS4AFA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.638370352 (596h35m14.509453397s) [noise=true , address=145.79.227.14:4856 , piece_id=INFPJOZKP2RRKA2J5WIGO4ZGETE2O3POMI7QOQFZGI2L7WUQE4BQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.699383661 (596h35m14.570466696s) [noise=true , address=145.79.227.18:4026 , piece_id=2OUCERATQ2SPA4AFQ5IXIUKQL5EUBF77M7ULPEONMXJ3S3VUK4KQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.862937723 (596h35m14.734020758s) [noise=true , address=77.90.31.14:4678 , piece_id=EIJW64GI4JNKRCLPQWPRGI4N4E4ZPXAM6HRQRNNBRZCTVKDJIMYA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.914517709 (596h35m14.785600754s) [noise=true , address=77.90.31.19:4862 , piece_id=3KXQYBZX2IEKBPGMBBASXG4PZSL5VKPJQC4OG32QZKCJC3XTWWIA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:14.986085857 (596h35m14.857168892s) [noise=true , address=45.91.116.4:4971 , piece_id=N6RJTJ4O3APBQYFLFYSS5YWHNXFKZI4JWYUP35WSEZESFGVHJMGQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.055462621 (596h35m14.926545656s) [noise=true , address=45.91.116.10:4095 , piece_id=AJSV4HFS4YOLXA4NCPQNZGMGWE3R6BHWYDK572EZ3VBBYF7IFFRQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.540782225 (596h35m15.41186526s) [noise=true , address=77.90.31.12:4600 , piece_id=FUEZGUXFX3T67EGDSRIVMT6JOKKFGJPO3MEZXSUIEATTHBQMUCRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.566199041 (596h35m15.437282076s) [noise=true , address=145.79.227.44:4202 , piece_id=VYTZDIEURQ6VUR4DVOXQDQTFD5JL2VO57IKNDPCCHJOMGIIE3XIQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.576173684 (596h35m15.447256729s) [noise=true , address=145.79.227.30:4776 , piece_id=RALYPZF7RU57XKRODHZ5XUQWMHJC6RCFDJOKM6AJ6LQBEQCFMUVQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.609003518 (596h35m15.480086553s) [noise=true , address=193.149.164.143:4704 , piece_id=IUWNLTD6L3F57QMPCGXX3SSWT55EXAVE64DKOS5M2KY4T6RUWMQA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.636248797 (596h35m15.507331842s) [noise=true , address=145.79.227.35:4515 , piece_id=MWN4KMHOWLD37K6VMDWCF5WVJHYZUUIGWDQ26DNBMJWCIVFTAJUQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.686242744 (596h35m15.557325799s) [noise=true , address=145.79.227.33:4888 , piece_id=WUN55K554NH5VQEPIL7LSOFH6RBKUOD4PNG2ZAPQVWQW43SQT7CQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701215324 (596h35m15.572298369s) [noise=true , address=145.79.227.40:4075 , piece_id=6FWGP2PMUVD23L3BPWCM2VKEKZUHQOGAKZXR3POOZAQHD4IMUDJQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: <nil> Dec 30 12:04:15.701242094 (596h35m15.57232517s) [seg_pos={0 0}, upload=522962]: Segment reached optimal threshold of 14 pieces. Dec 30 12:04:15.701481861 (596h35m15.572564906s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701493063 (596h35m15.572576108s) [noise=true , address=77.90.31.15:4711 , piece_id=KJNZASNRMV4G3XKIKC5DAGZJPHAJJADRM6AM5U5OFVA56MXNW6PA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701506388 (596h35m15.572589433s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36944->64.49.13.169:4707: use of closed network connection Dec 30 12:04:15.701511347 (596h35m15.572594382s) [noise=true , address=64.49.13.169:4707 , piece_id=RK5NIUAN7IWQC4SEBH5JX2FI2I2BP2LLU4APD6MM4CPF3AZ2UMEQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701634883 (596h35m15.572717928s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701641866 (596h35m15.572724911s) [noise=true , address=145.79.227.49:4361 , piece_id=A6D4QWQKF5MCKSBTI5JBZBVOP2BBM2AYEWRANKTBRO2LIXZDYNRA, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701701380 (596h35m15.572784415s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hdE48UnygY1dCRASR39FkumY2V8f5Xb68Pd1WJ4AMLUnnGAfT): protocol: expected piece hash; context canceled; EOF Dec 30 12:04:15.701711088 (596h35m15.572794123s) [noise=true , address=64.49.13.26:4946 , piece_id=SGMGI4HU6H3FRFZ5VVYM6HU64E2575D4KAXI3OJ2MBRVMLUKU7OQ, seg_pos={0 0}, upload=522962]: Long tail cancellation due to optimal threshold Dec 30 12:04:15.701891792 (596h35m15.572974837s) [seg_pos={0 0}, upload=522962]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 12:04:15.701991632 (596h35m15.573074677s) [upload=522962]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 12:04:15.701996742 (596h35m15.573079777s) [upload=522962]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 12:04:15.992115763 (596h35m15.863198798s) [upload=522962]: Done uploading segments... Dec 30 12:04:15.992134419 (596h35m15.863217464s) [upload=522962]: Upload finished. err: <nil> Duration: 40.51583462s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)