Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.02036593s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:37:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.020393662s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.592738573s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:36:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.592764623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 20.578337529s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:36:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 20.578364791s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.74258681s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:36:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.742611676s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.611052288s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:35:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.611083798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 16.71233921s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:35:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 16.712371561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.513448056s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:35:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.513483262s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.56555049s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:35:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.565581842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 15.155734158s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:34:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 15.15575044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.659403917s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:34:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.659434357s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.092668812s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:34:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.0927041s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 23.609977928s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:34:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 23.610022864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.687057108s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:33:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.687083858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.645506859s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:33:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.645540093s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 13.151688117s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:33:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 13.151717473s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.621726242s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:32:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.621753403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.661119204s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:32:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.661146075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 18.026017218s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:32:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 18.026045703s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.145704071s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:32:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.145731504s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.904602437s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:31:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.904627164s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 28.399878222s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:31:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 28.399906796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.484172627s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:31:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484197375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.612823853s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.612856375s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.53137642s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:30:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531402109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 18.845570653s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:30:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 18.845590591s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 4.467610534s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:30:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.467640121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 25.503845015s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:30:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 25.503866645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.635265869s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:30:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.635294322s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 23.474600875s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:29:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 23.474624699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.59584788s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:29:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.595907182s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.05139316s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:28:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.051451782s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 17.417491372s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:28:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 17.417518575s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.829644807s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:28:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.82969934s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.494684805s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:28:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494710604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.247204016s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:27:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.247232251s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.58138078s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:27:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.581402029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 10.177161064s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:27:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 10.177204647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.808599041s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:26:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.808627925s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.376735767s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:26:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.376762228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 14.848380357s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:26:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 14.8484089s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.603263989s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:26:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.603295158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.743223939s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:26:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.743253156s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.998603473s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:25:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.99861811s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 12.361543216s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:25:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 12.361571631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 28.666138098s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:25:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 28.666157925s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.49955452s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:25:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499589617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.615892959s trace: https://jaeger.dev.storj.io/trace/3251959fa0d27548 FAIL (2025-12-16 07:24:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.615908298s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 26.517858607s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:24:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 26.517888934s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.428015014s trace: https://jaeger.dev.storj.io/trace/7d4279450e3d7fc3 FAIL (2025-12-16 07:24:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428045522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 35.805142086s trace: https://jaeger.dev.storj.io/trace/76b7a1d8e48b6a48 FAIL (2025-12-16 07:24:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nub1MZAu4tZjy9K4WdDvAPABWt63Zvf942uqGQnYTPgUKYiQ2, Piece ID: C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA) context deadline exceeded; piecestore: (Node ID: 121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, Piece ID: 4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ) context deadline exceeded; piecestore: (Node ID: 15C8cVb1V9mFSFvXb2ypahffPKKMsAvQ6JWMhe8s113kmwYMVk, Piece ID: MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA) context deadline exceeded ======LOGS====== h53m2.04024479s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169445696 (255h53m2.040528731s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.169726531 (255h53m2.040809576s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170042773 (255h53m2.041125818s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.170582881 (255h53m2.041665926s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.171051163 (255h53m2.042134208s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173524553 (255h53m2.044607598s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.173731106 (255h53m2.044814152s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.176376283 (255h53m2.047459328s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.180291338 (255h53m2.051374373s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.186582337 (255h53m2.057665382s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.189866440 (255h53m2.060949485s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.191757732 (255h53m2.062840777s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:02.195321438 (255h53m2.066404483s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Uploading piece... Dec 16 07:22:03.659222643 (255h53m3.530305678s) [upload=224356]: Next returned nil segment Dec 16 07:22:03.659238755 (255h53m3.53032179s) [upload=224356]: Waiting for error group managing segments... Dec 16 07:22:04.057128554 (255h53m3.928211609s) [noise=true , address=145.79.227.43:4183 , piece_id=MQZMUVRRZH7FWSGMA5O3KN72GUKHONOKTTTN6UA4OHSX4CNDPO2Q, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.111548731 (255h53m3.982631776s) [noise=true , address=5.231.73.5:4014 , piece_id=MCRGLI7LCTJIGN4LPLE2M2FYGBOFJNSCOY4XQOMNZLSE65AUZXTA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.171755027 (255h53m4.042838072s) [noise=true , address=5.231.73.4:4009 , piece_id=C5XGN4TMZH45NOIJKFAXKVBU3JB3QDJNLGAUENDUCNOYXBJ2UHOA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.235552249 (255h53m4.106635294s) [noise=true , address=77.90.31.12:4616 , piece_id=WLTMMPRAFOZEPSHCKGUKV3DNC35GAGM64QBS3YWYLL43P6OTJ7JQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.248299793 (255h53m4.119382838s) [noise=true , address=5.231.73.2:4014 , piece_id=GMB4IXFNQGWQDATCZGK7YHTYTE63DCQA4M7HMEHSUCKML2DTYOYA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.272059846 (255h53m4.143142891s) [noise=true , address=77.90.31.13:4647 , piece_id=4BFMVI5EDXS6O6YUMYRSQXVV25XL3UKOUUG4JFHW6IVGE5N6PIMQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.284326535 (255h53m4.15540958s) [noise=true , address=77.90.31.18:4831 , piece_id=J5D5M5J3YOIP6ADST3W4AHAP2HOT3VJ6UYCPPEGXT6FGCWDQK3AQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.285474391 (255h53m4.156557436s) [noise=true , address=77.90.31.21:4885 , piece_id=LIVMB7HJVMLSCHUKVNVQQWJHFORY4C67ZLOBEWN74UPG4QJD6ZXQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.302511372 (255h53m4.173594418s) [noise=true , address=77.90.31.11:4586 , piece_id=MLMYF4XY3D4H2ECXPAR2VGVCOV3RQ3J2IC5PVKMXJ6ZBXOZIHIKA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.304610068 (255h53m4.175693114s) [noise=true , address=77.90.31.16:4766 , piece_id=GRUOBAT7HZ2IR7R4D3Z4Y6KA4PNBOC7L5LGYWHTKZARYEOXZJ5ZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:04.359237961 (255h53m4.230321006s) [noise=true , address=77.90.31.17:4801 , piece_id=3RY3RTLT7JOSXT273V75QOGB7FGPO5A26PLP2OVLX3K6FC67BDKQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:05.068555082 (255h53m4.939638127s) [noise=true , address=145.79.227.30:4790 , piece_id=WI2BODLORW42J4O7M5XQNX5J5YXQOAQZ5OFK3DDTM4SABSG4ZBBA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:17.318911161 (255h53m17.189994196s) [noise=true , address=64.49.13.169:4716 , piece_id=KQITPHEETWRUGETIDTAN3ZHCPKNAHT2JVQPFKKQFXKMD5UC25GEQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904210293 (255h53m53.775293328s) [noise=true , address=45.91.116.9:4278 , piece_id=O4CICJUY3NW4UX5Y4N6SEZNT2HGHPFMQJJRLVI3ZNJQQXD2V6THA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: <nil> Dec 16 07:22:53.904237565 (255h53m53.7753206s) [seg_pos={0 0}, upload=224356]: Segment reached optimal threshold of 14 pieces. Dec 16 07:22:53.904459547 (255h53m53.775542592s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY): protocol: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46648->45.91.116.5:4310: use of closed network connection Dec 16 07:22:53.904472302 (255h53m53.775555357s) [noise=true , address=45.91.116.5:4310 , piece_id=RQ34HIFGTKN4AVMIKLUAQHDEIZUCLHIEWAMVFG4IZ5A5SU52X3GQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.904819102 (255h53m53.775902157s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RTf8RiqgMBMmVth9UwPVs2DSYE5gQUz59tGvQebr2xFui4wXK): protocol: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43264->45.91.116.10:4090: use of closed network connection Dec 16 07:22:53.904827959 (255h53m53.775911004s) [noise=true , address=45.91.116.10:4090 , piece_id=63XGFEXYUAJ3FDDYWB24FH2J7I4SKFXXXB7IRDZDVUVE2JU6CKZQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905067856 (255h53m53.776150901s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1): protocol: rpc: dial tcp 45.91.116.13:4173: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905075420 (255h53m53.776158465s) [noise=true , address=45.91.116.13:4173 , piece_id=7FDBIEQIQ26SHBQHLHNEKQ3ISHWTC2EPILXTFMZZ24HOUPAQIANQ, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905246837 (255h53m53.776329892s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi): protocol: read tcp 188.166.225.170:60058->45.91.116.7:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 16 07:22:53.905255643 (255h53m53.776338678s) [noise=true , address=45.91.116.7:4022 , piece_id=HL2F52KBRNGEN3RITMGIRLTOMFFKJK7DCSICAU5HVQUUDZOINWDA, seg_pos={0 0}, upload=224356]: Long tail cancellation due to optimal threshold Dec 16 07:22:53.905281633 (255h53m53.776364678s) [seg_pos={0 0}, upload=224356]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 07:22:53.905362376 (255h53m53.776445411s) [upload=224356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 07:22:53.905368799 (255h53m53.776451844s) [upload=224356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 07:22:54.213006402 (255h53m54.084089437s) [upload=224356]: Done uploading segments... Dec 16 07:22:54.213027411 (255h53m54.084110446s) [upload=224356]: Upload finished. err: <nil> Duration: 35.805158809s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)