Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X...........] duration: 3.412230298s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:10:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.412256609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X....X.XX..X....XX.XX..................X...X..X..X..XX.X.X..] duration: 6.24880616s trace: https://jaeger.dev.storj.io/trace/3091684ebdfebfaf FAIL (2025-12-30 08:10:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ) context deadline exceeded ======LOGS====== egments... Dec 30 08:08:56.906787962 (592h39m56.777871007s) [upload=519577]: Got next segment. Inline: false Dec 30 08:08:56.906835392 (592h39m56.777918427s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:08:56.906849899 (592h39m56.777932945s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:08:57.151984305 (592h39m57.02306735s) [seg_pos={0 0}, upload=519577]: Begin upload segment... Dec 30 08:08:57.152045421 (592h39m57.023128466s) [seg_pos={0 0}, upload=519577]: Done begin upload segment. Dec 30 08:08:57.152088784 (592h39m57.023171829s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152103782 (592h39m57.023186817s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152948079 (592h39m57.024031114s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153318124 (592h39m57.024401159s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153610311 (592h39m57.024693346s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154041302 (592h39m57.025124337s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154116154 (592h39m57.025199199s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154499845 (592h39m57.02558289s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154851113 (592h39m57.025934158s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155328523 (592h39m57.026411558s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155630638 (592h39m57.026713673s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156234418 (592h39m57.027317463s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156721465 (592h39m57.02780451s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.158361338 (592h39m57.029444383s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.159690889 (592h39m57.030773924s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.161372472 (592h39m57.032455517s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:58.851873092 (592h39m58.722956137s) [upload=519577]: Next returned nil segment Dec 30 08:08:58.851904202 (592h39m58.722987247s) [upload=519577]: Waiting for error group managing segments... Dec 30 08:08:59.143407059 (592h39m59.014490104s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.143602201 (592h39m59.014685246s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.144760817 (592h39m59.015843852s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.153552237 (592h39m59.024635272s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.173075132 (592h39m59.044158177s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.230104896 (592h39m59.101187942s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238196233 (592h39m59.109279268s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238504911 (592h39m59.109587946s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.325614672 (592h39m59.196697717s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.439682385 (592h39m59.31076542s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.445870789 (592h39m59.316953854s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.467867354 (592h39m59.338950399s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494552707 (592h39m59.365635742s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494573135 (592h39m59.36565617s) [seg_pos={0 0}, upload=519577]: Segment reached optimal threshold of 13 pieces. Dec 30 08:08:59.494810718 (592h39m59.365893753s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection Dec 30 08:08:59.494817400 (592h39m59.365900436s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.494989989 (592h39m59.366073024s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection Dec 30 08:08:59.495013414 (592h39m59.366096459s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495030105 (592h39m59.36611314s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 08:08:59.495036769 (592h39m59.366119804s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495107033 (592h39m59.366190068s) [seg_pos={0 0}, upload=519577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:08:59.495174291 (592h39m59.366257326s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:08:59.495179631 (592h39m59.366262666s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:08:59.793154332 (592h39m59.664237367s) [upload=519577]: Done uploading segments... Dec 30 08:08:59.793190511 (592h39m59.664273546s) [upload=519577]: Upload finished. err: <nil> Duration: 6.248835626s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..........] duration: 3.547848286s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:10:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547884135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X....X.XX..X....XX.XX..................X...X..X..X..XX.X.X.] duration: 5.832260519s trace: https://jaeger.dev.storj.io/trace/3091684ebdfebfaf FAIL (2025-12-30 08:10:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ) context deadline exceeded ======LOGS====== egments... Dec 30 08:08:56.906787962 (592h39m56.777871007s) [upload=519577]: Got next segment. Inline: false Dec 30 08:08:56.906835392 (592h39m56.777918427s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:08:56.906849899 (592h39m56.777932945s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:08:57.151984305 (592h39m57.02306735s) [seg_pos={0 0}, upload=519577]: Begin upload segment... Dec 30 08:08:57.152045421 (592h39m57.023128466s) [seg_pos={0 0}, upload=519577]: Done begin upload segment. Dec 30 08:08:57.152088784 (592h39m57.023171829s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152103782 (592h39m57.023186817s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152948079 (592h39m57.024031114s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153318124 (592h39m57.024401159s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153610311 (592h39m57.024693346s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154041302 (592h39m57.025124337s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154116154 (592h39m57.025199199s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154499845 (592h39m57.02558289s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154851113 (592h39m57.025934158s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155328523 (592h39m57.026411558s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155630638 (592h39m57.026713673s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156234418 (592h39m57.027317463s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156721465 (592h39m57.02780451s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.158361338 (592h39m57.029444383s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.159690889 (592h39m57.030773924s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.161372472 (592h39m57.032455517s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:58.851873092 (592h39m58.722956137s) [upload=519577]: Next returned nil segment Dec 30 08:08:58.851904202 (592h39m58.722987247s) [upload=519577]: Waiting for error group managing segments... Dec 30 08:08:59.143407059 (592h39m59.014490104s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.143602201 (592h39m59.014685246s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.144760817 (592h39m59.015843852s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.153552237 (592h39m59.024635272s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.173075132 (592h39m59.044158177s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.230104896 (592h39m59.101187942s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238196233 (592h39m59.109279268s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238504911 (592h39m59.109587946s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.325614672 (592h39m59.196697717s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.439682385 (592h39m59.31076542s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.445870789 (592h39m59.316953854s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.467867354 (592h39m59.338950399s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494552707 (592h39m59.365635742s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494573135 (592h39m59.36565617s) [seg_pos={0 0}, upload=519577]: Segment reached optimal threshold of 13 pieces. Dec 30 08:08:59.494810718 (592h39m59.365893753s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection Dec 30 08:08:59.494817400 (592h39m59.365900436s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.494989989 (592h39m59.366073024s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection Dec 30 08:08:59.495013414 (592h39m59.366096459s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495030105 (592h39m59.36611314s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 08:08:59.495036769 (592h39m59.366119804s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495107033 (592h39m59.366190068s) [seg_pos={0 0}, upload=519577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:08:59.495174291 (592h39m59.366257326s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:08:59.495179631 (592h39m59.366262666s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:08:59.793154332 (592h39m59.664237367s) [upload=519577]: Done uploading segments... Dec 30 08:08:59.793190511 (592h39m59.664273546s) [upload=519577]: Upload finished. err: <nil> Duration: 5.832307888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X................X....] duration: 15.126082533s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:09:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 15.126110316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X....X.XX..X....XX.XX..................X...X..X..X..XX.X.X] duration: 30.002819579s trace: https://jaeger.dev.storj.io/trace/3091684ebdfebfaf FAIL (2025-12-30 08:09:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ) context deadline exceeded ======LOGS====== egments... Dec 30 08:08:56.906787962 (592h39m56.777871007s) [upload=519577]: Got next segment. Inline: false Dec 30 08:08:56.906835392 (592h39m56.777918427s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:08:56.906849899 (592h39m56.777932945s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:08:57.151984305 (592h39m57.02306735s) [seg_pos={0 0}, upload=519577]: Begin upload segment... Dec 30 08:08:57.152045421 (592h39m57.023128466s) [seg_pos={0 0}, upload=519577]: Done begin upload segment. Dec 30 08:08:57.152088784 (592h39m57.023171829s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152103782 (592h39m57.023186817s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.152948079 (592h39m57.024031114s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153318124 (592h39m57.024401159s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.153610311 (592h39m57.024693346s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154041302 (592h39m57.025124337s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154116154 (592h39m57.025199199s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154499845 (592h39m57.02558289s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.154851113 (592h39m57.025934158s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155328523 (592h39m57.026411558s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.155630638 (592h39m57.026713673s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156234418 (592h39m57.027317463s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.156721465 (592h39m57.02780451s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.158361338 (592h39m57.029444383s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.159690889 (592h39m57.030773924s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:57.161372472 (592h39m57.032455517s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Uploading piece... Dec 30 08:08:58.851873092 (592h39m58.722956137s) [upload=519577]: Next returned nil segment Dec 30 08:08:58.851904202 (592h39m58.722987247s) [upload=519577]: Waiting for error group managing segments... Dec 30 08:08:59.143407059 (592h39m59.014490104s) [noise=true , address=145.79.227.9:4366 , piece_id=K2DRU72BUKEPEXH7LOV6ALARBBJJKPTNBUUNO64UB3BXTS55PGOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.143602201 (592h39m59.014685246s) [noise=true , address=145.79.227.12:4765 , piece_id=CGKRA4XO6X67JNPKCJVO5QL7IGRWZOOJNY3JUBSCABOWFDQ5D6XA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.144760817 (592h39m59.015843852s) [noise=true , address=145.79.227.5:4223 , piece_id=KT7KHFMKSEBX336N4VMF56JH2K7F6ROLZYSQEP7SHJHER4WK6ZXA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.153552237 (592h39m59.024635272s) [noise=true , address=145.79.227.17:4972 , piece_id=J5KBR6FHEH77E7HY5PH6VJ6KMEVSOZNOZSGN7BAKNKBWPLMJ57WQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.173075132 (592h39m59.044158177s) [noise=true , address=145.79.227.23:4547 , piece_id=EDM6G77NYBVDAYEW35JCX7GZXZLT7PIRUMOQJKYVQENQWCLN6JUA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.230104896 (592h39m59.101187942s) [noise=true , address=145.79.227.16:4911 , piece_id=QVQGPEV4CMDLIDLVX4MLGWWRYDWTD5MPAHDKI4DA6XRZHLRA32FQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238196233 (592h39m59.109279268s) [noise=true , address=145.79.227.11:4732 , piece_id=KIZ35GQR4CGSDVADEDSACGTNQN6Z7TFCCB3XOCJMQJGJJOMN2GBA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.238504911 (592h39m59.109587946s) [noise=true , address=145.79.227.28:4727 , piece_id=2CN6VXXEZJMPDS56YUE5BUBALUOZQJXYCI7VAOBFCTZ64ZJUQRJQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.325614672 (592h39m59.196697717s) [noise=true , address=145.79.227.49:4337 , piece_id=5IMHTAFWPGWJ4EGNHAHKSKH5TDVQEGX5EREFHCDXX7O7C7SSNZ5A, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.439682385 (592h39m59.31076542s) [noise=true , address=45.91.116.9:4263 , piece_id=KKLW6YCZ5F3IEHCKCVAU2ZPFRL5K7KACPHCRK7STV7MMFU6ELSDQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.445870789 (592h39m59.316953854s) [noise=true , address=45.91.116.11:4119 , piece_id=YGPRJY2MM5NPPHWK5BJW6NZXBB4RAI432E2NJMFP6GGRD3HABAEQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.467867354 (592h39m59.338950399s) [noise=true , address=45.91.116.8:4047 , piece_id=NZ6TRDEY7SDEDW73ORQSAA4ADF2GH2CPCOKHU2JLHAV7GOVHFHQQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494552707 (592h39m59.365635742s) [noise=true , address=45.91.116.10:4081 , piece_id=VL5T6WF6KTVLOG3KTGXA7S766GPDWSP5AKCEX75YN2VOFQOYAZVQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: <nil> Dec 30 08:08:59.494573135 (592h39m59.36565617s) [seg_pos={0 0}, upload=519577]: Segment reached optimal threshold of 13 pieces. Dec 30 08:08:59.494810718 (592h39m59.365893753s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56574->77.90.31.10:4519: use of closed network connection Dec 30 08:08:59.494817400 (592h39m59.365900436s) [noise=true , address=77.90.31.10:4519 , piece_id=474U6E5HFO3MMVIU4TG3KVBFFBG244C3HUCT6NJ5KUBR2CS2SPCA, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.494989989 (592h39m59.366073024s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38694->45.91.116.14:4191: use of closed network connection Dec 30 08:08:59.495013414 (592h39m59.366096459s) [noise=true , address=45.91.116.14:4191 , piece_id=SYL2LKKFOMZH5YUCIEFYJWDHZZ7OMMTM2JOFXPRRKYOWKF6NQKOQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495030105 (592h39m59.36611314s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 08:08:59.495036769 (592h39m59.366119804s) [noise=true , address=45.91.116.4:4937 , piece_id=4PJFRBLLYDWEIVCO2PPPVIUVLIXSKSRS4EOU7JHHN6Q764RXC4ZQ, seg_pos={0 0}, upload=519577]: Long tail cancellation due to optimal threshold Dec 30 08:08:59.495107033 (592h39m59.366190068s) [seg_pos={0 0}, upload=519577]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:08:59.495174291 (592h39m59.366257326s) [upload=519577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:08:59.495179631 (592h39m59.366262666s) [upload=519577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:08:59.793154332 (592h39m59.664237367s) [upload=519577]: Done uploading segments... Dec 30 08:08:59.793190511 (592h39m59.664273546s) [upload=519577]: Upload finished. err: <nil> Duration: 30.002844686s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X.........] duration: 3.414705204s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:09:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.414726113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X................X...] duration: 15.415652269s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:09:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 15.415667287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X................X..] duration: 39.345705382s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:08:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 39.345738816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X....X.XX..X....XX.XX..................X...X..X..X..XX.X.] duration: 6.368882085s trace: https://jaeger.dev.storj.io/trace/6b18e7c223489e3f FAIL (2025-12-30 08:08:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 08:07:21.255268592 (592h38m21.126351647s) [upload=519551]: Starting upload Dec 30 08:07:21.255339848 (592h38m21.126422883s) [upload=519551]: Done starting upload Dec 30 08:07:21.255414541 (592h38m21.126497586s) [upload=519551]: Uploading segments... Dec 30 08:07:21.255648666 (592h38m21.126731711s) [upload=519551]: Got next segment. Inline: false Dec 30 08:07:21.255788542 (592h38m21.126871587s) [upload=519551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:07:21.255796949 (592h38m21.126879994s) [upload=519551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:07:21.503376400 (592h38m21.374459435s) [seg_pos={0 0}, upload=519551]: Begin upload segment... Dec 30 08:07:21.503438427 (592h38m21.374521472s) [seg_pos={0 0}, upload=519551]: Done begin upload segment. Dec 30 08:07:21.503473424 (592h38m21.374556469s) [noise=true , address=145.79.227.2:4115 , piece_id=X2JHLZ5NG4EUMVRVJURX6PDF26DF7H3HEUDCJDLZ5JIL273EKD3A, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.503485938 (592h38m21.374568973s) [noise=true , address=145.79.227.29:4941 , piece_id=P25E643UODVQC3AKCD5HWAPCFD43UI7Z664KAHGSTP5IXNBPUN6A, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.504149381 (592h38m21.375232427s) [noise=true , address=145.79.227.38:4978 , piece_id=2NGHIA3R5V7QI2AG3MPSQ37DDV576YGAFAM3SXR3EZPB55HFXFRQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.504267576 (592h38m21.375350621s) [noise=true , address=77.90.31.11:4555 , piece_id=VMY2T2UUSFU3UV3UHENVNFSFIZJKFJ7MPW4QPG4WCJD5I547SLCQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.504663731 (592h38m21.375746776s) [noise=true , address=77.90.31.18:4805 , piece_id=U2B62AOLUW56H4AO3JOZXXEZQAQJJD75FO4HRZPAKICEUDJNJXWQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.504674461 (592h38m21.375757506s) [noise=true , address=77.90.31.14:4661 , piece_id=DIHVGUWJNAFSXE32IMAVJAEXX4OLSK3HOF25W2QMMYS4EYDAU5OQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.505122294 (592h38m21.376205329s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QXIH7OHYPUKLKG5CGGBK4NP2T7KM7BHQRQFOLASVD64MY7RFTA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.505322215 (592h38m21.37640526s) [noise=true , address=145.79.227.50:4373 , piece_id=G53PGY6KUUL6V5CINRPGH2OXEWROYVLQCTZV55N7GLCMGLVVK5JA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.505603340 (592h38m21.376686385s) [noise=true , address=145.79.227.22:4510 , piece_id=YPFUY7WVMJB3VSBQJY2J5NOQMCB6B47MGUJGQSDWZCNMXRHTAYQA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.506368247 (592h38m21.377451292s) [noise=true , address=145.79.227.26:4654 , piece_id=YERPRKDLMZJZYRI4S624YWCEKK5QDM377RWM2BHRAJXHHL2UH2TQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.506663418 (592h38m21.377746464s) [noise=true , address=45.91.116.5:4300 , piece_id=R5AQFMRBJ572F5PJJRJJA3XCGJ22IHNB7WV4ZE3ZHKGXNVEV5DYA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.507376226 (592h38m21.378459271s) [noise=true , address=145.79.227.48:4301 , piece_id=I3WYJ2RH4HE2JXRUKAJQDHPP46NNNA4AWM5J6YJZXYLT62JK3ROA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.508034700 (592h38m21.379117745s) [noise=true , address=45.91.116.4:4939 , piece_id=6L2H23HW2YSPHFLHUSNTK7TYPHOGS7Y5ZLXCLEUM6V3FZL3TFSGA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.508868136 (592h38m21.379951181s) [noise=true , address=145.79.227.8:4329 , piece_id=AIHQSLHD4MJNLTMFCSQQ5TGF2OYCJHH64FU3FZAQUWUX4BAPB4PA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.509428314 (592h38m21.380511359s) [noise=true , address=45.91.116.11:4119 , piece_id=V4NOXQ2SBZ2XWRGOVGRO47THVJAYP3DP7MUYXVFGUPYFDCRS52DQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.510276228 (592h38m21.381359273s) [noise=true , address=145.79.227.12:4765 , piece_id=HIQKLIKYH3FW42GPUQI4U574QPZE4AZYAJK5YRQQ2NDXFTGX3QIA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:23.185205789 (592h38m23.056288824s) [upload=519551]: Next returned nil segment Dec 30 08:07:23.185231718 (592h38m23.056314754s) [upload=519551]: Waiting for error group managing segments... Dec 30 08:07:23.500199247 (592h38m23.371282292s) [noise=true , address=145.79.227.8:4329 , piece_id=AIHQSLHD4MJNLTMFCSQQ5TGF2OYCJHH64FU3FZAQUWUX4BAPB4PA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.515998992 (592h38m23.387082027s) [noise=true , address=145.79.227.2:4115 , piece_id=X2JHLZ5NG4EUMVRVJURX6PDF26DF7H3HEUDCJDLZ5JIL273EKD3A, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.517338964 (592h38m23.388422009s) [noise=true , address=145.79.227.38:4978 , piece_id=2NGHIA3R5V7QI2AG3MPSQ37DDV576YGAFAM3SXR3EZPB55HFXFRQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.519768499 (592h38m23.390851544s) [noise=true , address=145.79.227.50:4373 , piece_id=G53PGY6KUUL6V5CINRPGH2OXEWROYVLQCTZV55N7GLCMGLVVK5JA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.521017698 (592h38m23.392100743s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QXIH7OHYPUKLKG5CGGBK4NP2T7KM7BHQRQFOLASVD64MY7RFTA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.527685034 (592h38m23.398768069s) [noise=true , address=145.79.227.12:4765 , piece_id=HIQKLIKYH3FW42GPUQI4U574QPZE4AZYAJK5YRQQ2NDXFTGX3QIA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.529717684 (592h38m23.400800729s) [noise=true , address=145.79.227.48:4301 , piece_id=I3WYJ2RH4HE2JXRUKAJQDHPP46NNNA4AWM5J6YJZXYLT62JK3ROA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.589779111 (592h38m23.460862156s) [noise=true , address=145.79.227.29:4941 , piece_id=P25E643UODVQC3AKCD5HWAPCFD43UI7Z664KAHGSTP5IXNBPUN6A, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.593390057 (592h38m23.464473102s) [noise=true , address=145.79.227.26:4654 , piece_id=YERPRKDLMZJZYRI4S624YWCEKK5QDM377RWM2BHRAJXHHL2UH2TQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.660299653 (592h38m23.531382688s) [noise=true , address=145.79.227.22:4510 , piece_id=YPFUY7WVMJB3VSBQJY2J5NOQMCB6B47MGUJGQSDWZCNMXRHTAYQA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.668519655 (592h38m23.53960269s) [noise=true , address=77.90.31.18:4805 , piece_id=U2B62AOLUW56H4AO3JOZXXEZQAQJJD75FO4HRZPAKICEUDJNJXWQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.680581403 (592h38m23.551664438s) [noise=true , address=77.90.31.11:4555 , piece_id=VMY2T2UUSFU3UV3UHENVNFSFIZJKFJ7MPW4QPG4WCJD5I547SLCQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.869187209 (592h38m23.740270254s) [noise=true , address=45.91.116.5:4300 , piece_id=R5AQFMRBJ572F5PJJRJJA3XCGJ22IHNB7WV4ZE3ZHKGXNVEV5DYA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.869211014 (592h38m23.740294049s) [seg_pos={0 0}, upload=519551]: Segment reached optimal threshold of 13 pieces. Dec 30 08:07:23.869703361 (592h38m23.740786396s) [noise=true , address=45.91.116.11:4119 , piece_id=V4NOXQ2SBZ2XWRGOVGRO47THVJAYP3DP7MUYXVFGUPYFDCRS52DQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 08:07:23.869719993 (592h38m23.740803028s) [noise=true , address=45.91.116.11:4119 , piece_id=V4NOXQ2SBZ2XWRGOVGRO47THVJAYP3DP7MUYXVFGUPYFDCRS52DQ, seg_pos={0 0}, upload=519551]: Long tail cancellation due to optimal threshold Dec 30 08:07:23.869775057 (592h38m23.740858093s) [noise=true , address=77.90.31.14:4661 , piece_id=DIHVGUWJNAFSXE32IMAVJAEXX4OLSK3HOF25W2QMMYS4EYDAU5OQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 30 08:07:23.869782793 (592h38m23.740865848s) [noise=true , address=77.90.31.14:4661 , piece_id=DIHVGUWJNAFSXE32IMAVJAEXX4OLSK3HOF25W2QMMYS4EYDAU5OQ, seg_pos={0 0}, upload=519551]: Long tail cancellation due to optimal threshold Dec 30 08:07:23.869944510 (592h38m23.741027545s) [noise=true , address=45.91.116.4:4939 , piece_id=6L2H23HW2YSPHFLHUSNTK7TYPHOGS7Y5ZLXCLEUM6V3FZL3TFSGA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 08:07:23.869953859 (592h38m23.741036904s) [noise=true , address=45.91.116.4:4939 , piece_id=6L2H23HW2YSPHFLHUSNTK7TYPHOGS7Y5ZLXCLEUM6V3FZL3TFSGA, seg_pos={0 0}, upload=519551]: Long tail cancellation due to optimal threshold Dec 30 08:07:23.869986851 (592h38m23.741069896s) [seg_pos={0 0}, upload=519551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:07:23.870071913 (592h38m23.741154958s) [upload=519551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:07:23.870077534 (592h38m23.741160569s) [upload=519551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:07:24.171728956 (592h38m24.042812001s) [upload=519551]: Done uploading segments... Dec 30 08:07:24.171755296 (592h38m24.042838331s) [upload=519551]: Upload finished. err: <nil> Duration: 6.368920889s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X........] duration: 3.551169822s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:08:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551201182s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X....X.XX..X....XX.XX..................X...X..X..X..XX.X] duration: 30.003188708s trace: https://jaeger.dev.storj.io/trace/6b18e7c223489e3f FAIL (2025-12-30 08:07:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 08:07:21.255268592 (592h38m21.126351647s) [upload=519551]: Starting upload Dec 30 08:07:21.255339848 (592h38m21.126422883s) [upload=519551]: Done starting upload Dec 30 08:07:21.255414541 (592h38m21.126497586s) [upload=519551]: Uploading segments... Dec 30 08:07:21.255648666 (592h38m21.126731711s) [upload=519551]: Got next segment. Inline: false Dec 30 08:07:21.255788542 (592h38m21.126871587s) [upload=519551]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:07:21.255796949 (592h38m21.126879994s) [upload=519551]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:07:21.503376400 (592h38m21.374459435s) [seg_pos={0 0}, upload=519551]: Begin upload segment... Dec 30 08:07:21.503438427 (592h38m21.374521472s) [seg_pos={0 0}, upload=519551]: Done begin upload segment. Dec 30 08:07:21.503473424 (592h38m21.374556469s) [noise=true , address=145.79.227.2:4115 , piece_id=X2JHLZ5NG4EUMVRVJURX6PDF26DF7H3HEUDCJDLZ5JIL273EKD3A, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.503485938 (592h38m21.374568973s) [noise=true , address=145.79.227.29:4941 , piece_id=P25E643UODVQC3AKCD5HWAPCFD43UI7Z664KAHGSTP5IXNBPUN6A, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.504149381 (592h38m21.375232427s) [noise=true , address=145.79.227.38:4978 , piece_id=2NGHIA3R5V7QI2AG3MPSQ37DDV576YGAFAM3SXR3EZPB55HFXFRQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.504267576 (592h38m21.375350621s) [noise=true , address=77.90.31.11:4555 , piece_id=VMY2T2UUSFU3UV3UHENVNFSFIZJKFJ7MPW4QPG4WCJD5I547SLCQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.504663731 (592h38m21.375746776s) [noise=true , address=77.90.31.18:4805 , piece_id=U2B62AOLUW56H4AO3JOZXXEZQAQJJD75FO4HRZPAKICEUDJNJXWQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.504674461 (592h38m21.375757506s) [noise=true , address=77.90.31.14:4661 , piece_id=DIHVGUWJNAFSXE32IMAVJAEXX4OLSK3HOF25W2QMMYS4EYDAU5OQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.505122294 (592h38m21.376205329s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QXIH7OHYPUKLKG5CGGBK4NP2T7KM7BHQRQFOLASVD64MY7RFTA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.505322215 (592h38m21.37640526s) [noise=true , address=145.79.227.50:4373 , piece_id=G53PGY6KUUL6V5CINRPGH2OXEWROYVLQCTZV55N7GLCMGLVVK5JA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.505603340 (592h38m21.376686385s) [noise=true , address=145.79.227.22:4510 , piece_id=YPFUY7WVMJB3VSBQJY2J5NOQMCB6B47MGUJGQSDWZCNMXRHTAYQA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.506368247 (592h38m21.377451292s) [noise=true , address=145.79.227.26:4654 , piece_id=YERPRKDLMZJZYRI4S624YWCEKK5QDM377RWM2BHRAJXHHL2UH2TQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.506663418 (592h38m21.377746464s) [noise=true , address=45.91.116.5:4300 , piece_id=R5AQFMRBJ572F5PJJRJJA3XCGJ22IHNB7WV4ZE3ZHKGXNVEV5DYA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.507376226 (592h38m21.378459271s) [noise=true , address=145.79.227.48:4301 , piece_id=I3WYJ2RH4HE2JXRUKAJQDHPP46NNNA4AWM5J6YJZXYLT62JK3ROA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.508034700 (592h38m21.379117745s) [noise=true , address=45.91.116.4:4939 , piece_id=6L2H23HW2YSPHFLHUSNTK7TYPHOGS7Y5ZLXCLEUM6V3FZL3TFSGA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.508868136 (592h38m21.379951181s) [noise=true , address=145.79.227.8:4329 , piece_id=AIHQSLHD4MJNLTMFCSQQ5TGF2OYCJHH64FU3FZAQUWUX4BAPB4PA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.509428314 (592h38m21.380511359s) [noise=true , address=45.91.116.11:4119 , piece_id=V4NOXQ2SBZ2XWRGOVGRO47THVJAYP3DP7MUYXVFGUPYFDCRS52DQ, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:21.510276228 (592h38m21.381359273s) [noise=true , address=145.79.227.12:4765 , piece_id=HIQKLIKYH3FW42GPUQI4U574QPZE4AZYAJK5YRQQ2NDXFTGX3QIA, seg_pos={0 0}, upload=519551]: Uploading piece... Dec 30 08:07:23.185205789 (592h38m23.056288824s) [upload=519551]: Next returned nil segment Dec 30 08:07:23.185231718 (592h38m23.056314754s) [upload=519551]: Waiting for error group managing segments... Dec 30 08:07:23.500199247 (592h38m23.371282292s) [noise=true , address=145.79.227.8:4329 , piece_id=AIHQSLHD4MJNLTMFCSQQ5TGF2OYCJHH64FU3FZAQUWUX4BAPB4PA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.515998992 (592h38m23.387082027s) [noise=true , address=145.79.227.2:4115 , piece_id=X2JHLZ5NG4EUMVRVJURX6PDF26DF7H3HEUDCJDLZ5JIL273EKD3A, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.517338964 (592h38m23.388422009s) [noise=true , address=145.79.227.38:4978 , piece_id=2NGHIA3R5V7QI2AG3MPSQ37DDV576YGAFAM3SXR3EZPB55HFXFRQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.519768499 (592h38m23.390851544s) [noise=true , address=145.79.227.50:4373 , piece_id=G53PGY6KUUL6V5CINRPGH2OXEWROYVLQCTZV55N7GLCMGLVVK5JA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.521017698 (592h38m23.392100743s) [noise=true , address=145.79.227.27:4689 , piece_id=X4QXIH7OHYPUKLKG5CGGBK4NP2T7KM7BHQRQFOLASVD64MY7RFTA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.527685034 (592h38m23.398768069s) [noise=true , address=145.79.227.12:4765 , piece_id=HIQKLIKYH3FW42GPUQI4U574QPZE4AZYAJK5YRQQ2NDXFTGX3QIA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.529717684 (592h38m23.400800729s) [noise=true , address=145.79.227.48:4301 , piece_id=I3WYJ2RH4HE2JXRUKAJQDHPP46NNNA4AWM5J6YJZXYLT62JK3ROA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.589779111 (592h38m23.460862156s) [noise=true , address=145.79.227.29:4941 , piece_id=P25E643UODVQC3AKCD5HWAPCFD43UI7Z664KAHGSTP5IXNBPUN6A, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.593390057 (592h38m23.464473102s) [noise=true , address=145.79.227.26:4654 , piece_id=YERPRKDLMZJZYRI4S624YWCEKK5QDM377RWM2BHRAJXHHL2UH2TQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.660299653 (592h38m23.531382688s) [noise=true , address=145.79.227.22:4510 , piece_id=YPFUY7WVMJB3VSBQJY2J5NOQMCB6B47MGUJGQSDWZCNMXRHTAYQA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.668519655 (592h38m23.53960269s) [noise=true , address=77.90.31.18:4805 , piece_id=U2B62AOLUW56H4AO3JOZXXEZQAQJJD75FO4HRZPAKICEUDJNJXWQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.680581403 (592h38m23.551664438s) [noise=true , address=77.90.31.11:4555 , piece_id=VMY2T2UUSFU3UV3UHENVNFSFIZJKFJ7MPW4QPG4WCJD5I547SLCQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.869187209 (592h38m23.740270254s) [noise=true , address=45.91.116.5:4300 , piece_id=R5AQFMRBJ572F5PJJRJJA3XCGJ22IHNB7WV4ZE3ZHKGXNVEV5DYA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: <nil> Dec 30 08:07:23.869211014 (592h38m23.740294049s) [seg_pos={0 0}, upload=519551]: Segment reached optimal threshold of 13 pieces. Dec 30 08:07:23.869703361 (592h38m23.740786396s) [noise=true , address=45.91.116.11:4119 , piece_id=V4NOXQ2SBZ2XWRGOVGRO47THVJAYP3DP7MUYXVFGUPYFDCRS52DQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 08:07:23.869719993 (592h38m23.740803028s) [noise=true , address=45.91.116.11:4119 , piece_id=V4NOXQ2SBZ2XWRGOVGRO47THVJAYP3DP7MUYXVFGUPYFDCRS52DQ, seg_pos={0 0}, upload=519551]: Long tail cancellation due to optimal threshold Dec 30 08:07:23.869775057 (592h38m23.740858093s) [noise=true , address=77.90.31.14:4661 , piece_id=DIHVGUWJNAFSXE32IMAVJAEXX4OLSK3HOF25W2QMMYS4EYDAU5OQ, seg_pos={0 0}, upload=519551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 30 08:07:23.869782793 (592h38m23.740865848s) [noise=true , address=77.90.31.14:4661 , piece_id=DIHVGUWJNAFSXE32IMAVJAEXX4OLSK3HOF25W2QMMYS4EYDAU5OQ, seg_pos={0 0}, upload=519551]: Long tail cancellation due to optimal threshold Dec 30 08:07:23.869944510 (592h38m23.741027545s) [noise=true , address=45.91.116.4:4939 , piece_id=6L2H23HW2YSPHFLHUSNTK7TYPHOGS7Y5ZLXCLEUM6V3FZL3TFSGA, seg_pos={0 0}, upload=519551]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 08:07:23.869953859 (592h38m23.741036904s) [noise=true , address=45.91.116.4:4939 , piece_id=6L2H23HW2YSPHFLHUSNTK7TYPHOGS7Y5ZLXCLEUM6V3FZL3TFSGA, seg_pos={0 0}, upload=519551]: Long tail cancellation due to optimal threshold Dec 30 08:07:23.869986851 (592h38m23.741069896s) [seg_pos={0 0}, upload=519551]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:07:23.870071913 (592h38m23.741154958s) [upload=519551]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:07:23.870077534 (592h38m23.741160569s) [upload=519551]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:07:24.171728956 (592h38m24.042812001s) [upload=519551]: Done uploading segments... Dec 30 08:07:24.171755296 (592h38m24.042838331s) [upload=519551]: Upload finished. err: <nil> Duration: 30.00327952s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X................X.] duration: 29.787694583s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:07:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 29.787711004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X................X] duration: 1m0.004250556s trace: https://jaeger.dev.storj.io/trace/3368aa43178d2f5a FAIL (2025-12-30 08:07:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534536066 (592h37m18.405619101s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.534852999 (592h37m18.405936044s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535286856 (592h37m18.406369891s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.535707266 (592h37m18.406790301s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536251703 (592h37m18.407334738s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.536838330 (592h37m18.407921365s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.537331029 (592h37m18.408414064s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.538447985 (592h37m18.40953102s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.540364996 (592h37m18.411448041s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.542882459 (592h37m18.413965494s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.544737200 (592h37m18.415820235s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.546208583 (592h37m18.417291618s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:18.547919711 (592h37m18.419002746s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Uploading piece... Dec 30 08:06:19.971202137 (592h37m19.842285172s) [upload=519535]: Next returned nil segment Dec 30 08:06:19.971259255 (592h37m19.84234231s) [upload=519535]: Waiting for error group managing segments... Dec 30 08:06:20.403932505 (592h37m20.27501555s) [noise=true , address=145.79.227.7:4298 , piece_id=KWMK5C4K5DQ52VUE3SLEH4HXQ5A3OEP4LRHQTDRS2OLHTVH6IZXA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404571942 (592h37m20.275654977s) [noise=true , address=145.79.227.46:4604 , piece_id=INB3GL2L6MMSY2LXQQ7MPNI7EWIAAG3JWBTH7LL7EAEMWK2HV5PQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.404930254 (592h37m20.276013299s) [noise=true , address=64.49.13.27:4965 , piece_id=3SXMHPDDMARMFG45GWBGU4UDYFFSZJGIX47EKP3J27O6HPEI454Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.424140103 (592h37m20.295223148s) [noise=true , address=145.79.227.17:4982 , piece_id=TAQW6DFTISOIY75TOVUEHR23PTRQ5ONXMCQX6VA74NGMMAO7VWYA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.460339241 (592h37m20.331422286s) [noise=true , address=64.49.13.26:4949 , piece_id=SK6L3AI3YCCYAM2HYNYW24KAF7FOC4ILI5OP7VG74FQKWG4KJMCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.659134479 (592h37m20.530217514s) [noise=true , address=77.90.31.19:4856 , piece_id=CHCW6AFTGF47236IWBK2XUGPKFU5UB4MOEXDWFSFYHZ5ZZV5EHKQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.809729776 (592h37m20.680812822s) [noise=true , address=45.91.116.9:4289 , piece_id=BM2S7JE45W7NZI226EG4OPJMP3YS3LM764AHLHXUMG7V2AR3YUVA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:20.982444893 (592h37m20.853527928s) [noise=true , address=77.90.31.11:4576 , piece_id=KZJZ7LQZP7TOUX3V2BWXXN3AKA6YLOGZ7MJL4WXELII64IL4MMCQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.236558086 (592h37m21.107641131s) [noise=true , address=193.149.164.143:4697 , piece_id=FR2XTSS5SFT3HDQPKFDKKWII3Q2D4KWLTORGHCDBDVEBAHITNN7A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.280554351 (592h37m21.151637397s) [noise=true , address=193.149.164.143:4701 , piece_id=P24TST6QZF5LTDJZUDO77KWOER3MIYBR2SB7FLOWEMRKS7C3PTZQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.311838281 (592h37m21.182921336s) [noise=true , address=145.79.227.30:4766 , piece_id=KBQMPER57YYBH4EGP5RAXTDZZIQT2USXW3RS7PTLCKNFMD2FLRCA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.312983972 (592h37m21.184067008s) [noise=true , address=77.90.31.21:4902 , piece_id=4NDRTNSY2DKF5ZQEELA4BHLIM2V2P2ODEWOJ7O36UHPWXXAX7JJA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.329642783 (592h37m21.200725828s) [noise=true , address=145.79.227.44:4219 , piece_id=QVBP752V3F2BSL237AEVCZI2YO6DEDIOETPDCZRKL3O7IPJCGJ2A, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355443763 (592h37m21.226526798s) [noise=true , address=145.79.227.34:4935 , piece_id=ZF32R2OMRNW2EKIHHP3I6RSFSCJQSYP4QJYHVYWY7I7OG7L36RSA, seg_pos={0 0}, upload=519535]: Done uploading piece. err: <nil> Dec 30 08:06:21.355464893 (592h37m21.226547928s) [seg_pos={0 0}, upload=519535]: Segment reached optimal threshold of 14 pieces. Dec 30 08:06:21.355686715 (592h37m21.22676976s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->77.90.31.13:4654: use of closed network connection Dec 30 08:06:21.355697155 (592h37m21.2267802s) [noise=true , address=77.90.31.13:4654 , piece_id=NHLHJECGWLPBZATYUJ2SIUHFL26A534WMQ3FBPSJ55XVBTIOUWOQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355842501 (592h37m21.226925546s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49404->77.90.31.11:4573: use of closed network connection Dec 30 08:06:21.355874923 (592h37m21.226957968s) [noise=true , address=77.90.31.11:4573 , piece_id=USGPWSSULRIBJBCJKZ6LOCTVMWXXS6PIB7KGYKJOFDXSVKU5LRVQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.355899841 (592h37m21.226982916s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF): protocol: expected piece hash; context canceled; EOF Dec 30 08:06:21.355909068 (592h37m21.226992103s) [noise=true , address=145.79.227.43:4190 , piece_id=M3MCUMREXVHUQLL64TEE4TXYZMZIUCHZBEW7VR2D5MJTP7QNABXQ, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356025841 (592h37m21.227108876s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55458->64.49.13.169:4707: use of closed network connection Dec 30 08:06:21.356031792 (592h37m21.227114837s) [noise=true , address=64.49.13.169:4707 , piece_id=CP5Z6OEFPH553LEQE37QZTFPQ4NHQUN5F3VO5ZFY6ABCBVNGP67Q, seg_pos={0 0}, upload=519535]: Long tail cancellation due to optimal threshold Dec 30 08:06:21.356158484 (592h37m21.227241519s) [seg_pos={0 0}, upload=519535]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 08:06:21.356231643 (592h37m21.227314678s) [upload=519535]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:06:21.356236432 (592h37m21.227319477s) [upload=519535]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:06:21.659510163 (592h37m21.530593208s) [upload=519535]: Done uploading segments... Dec 30 08:06:21.659533387 (592h37m21.530616432s) [upload=519535]: Upload finished. err: <nil> Duration: 1m0.00427392s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X.......] duration: 3.420412177s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:07:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420441123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X....X....X.XX..X....XX.XX..................X...X..X..X..XX.] duration: 5.974539009s trace: https://jaeger.dev.storj.io/trace/71628c77d07a1aac FAIL (2025-12-30 08:07:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: T4Q5H657AXB2FOD3G2HOVNBFSGTJK4A4HNVYNCEYQQLPX53EMCOQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: OJPSDMAY26O64KW74R2DB6PFRQISHUTZZMRZP5DREGWVK4N3I4YQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NVGIUH7A7KSFLOGXHTYR3MS4EN6QFGFVE727BP52YW537ZOWJNRQ) context deadline exceeded ======LOGS====== 0 08:05:49.227438369 (592h36m49.098521414s) [upload=519528]: Starting upload Dec 30 08:05:49.227549210 (592h36m49.098632275s) [upload=519528]: Done starting upload Dec 30 08:05:49.227611589 (592h36m49.098694634s) [upload=519528]: Uploading segments... Dec 30 08:05:49.227925045 (592h36m49.099008091s) [upload=519528]: Got next segment. Inline: false Dec 30 08:05:49.227986202 (592h36m49.099069247s) [upload=519528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:05:49.227993105 (592h36m49.09907615s) [upload=519528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:05:49.474216123 (592h36m49.345299158s) [seg_pos={0 0}, upload=519528]: Begin upload segment... Dec 30 08:05:49.474271188 (592h36m49.345354223s) [seg_pos={0 0}, upload=519528]: Done begin upload segment. Dec 30 08:05:49.474337104 (592h36m49.345420149s) [noise=true , address=45.91.116.11:4120 , piece_id=3D4EBYSGHWBRWOTLVIBLHPCSZSAZ7H2HBE2256HFHVIRZVKWUYTA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.474399472 (592h36m49.345482507s) [noise=true , address=145.79.227.20:4438 , piece_id=7G5AQ2RA4S4VWCSWVMKJYZP7NRVFGSYBMZXIB37QDXPFJYH44OWA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.475006188 (592h36m49.346089223s) [noise=true , address=145.79.227.36:4519 , piece_id=NJ2AMF5VCXB2LK2ADGUQTOV74VOL7TWIVTZKUQZUJIENECJIDXWA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.475438661 (592h36m49.346521706s) [noise=true , address=145.79.227.40:4051 , piece_id=RBKROK3UFJ27GFE23EFUY5557W2OOK2MN5DJZQFUUFNF7DFLUESA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.475915770 (592h36m49.346998815s) [noise=true , address=145.79.227.10:4401 , piece_id=R752IMJI53EQ2L3B7WJY3FSWVPTHMRS4DHP6HH572F3E6I5JQQKQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.475926460 (592h36m49.347009505s) [noise=true , address=45.91.116.5:4300 , piece_id=NVGIUH7A7KSFLOGXHTYR3MS4EN6QFGFVE727BP52YW537ZOWJNRQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.476342642 (592h36m49.347425677s) [noise=true , address=77.90.31.19:4844 , piece_id=QS3RPY5XUJIBA4DTBGRJ2KPTNYH7V2UZK3NEG76YBRBWCVB7LS3A, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.476383560 (592h36m49.347466595s) [noise=true , address=77.90.31.21:4880 , piece_id=HWTZMJVYTPYAYFZ4R35LOY4BZ7EIDAK7FIPSQKNC7R4IBQKX25LA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.476799392 (592h36m49.347882427s) [noise=true , address=45.91.116.6:4976 , piece_id=VQAPJJXFX3F5FKDAYDOQJLO2JCULT2JM4N6NEHBJ4GEYVXDYOARQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.477105054 (592h36m49.348188089s) [noise=true , address=145.79.227.4:4187 , piece_id=JK2NEA6QVPEEEAKIPLTQZDLCQIIW4YJ7UN7HN55OPPSAB6TAUO5A, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.477403463 (592h36m49.348486498s) [noise=true , address=145.79.227.3:4150 , piece_id=MFGBL7CDWLDFR5RNIX36E64VRKAOFLZEMHLKUMDH6AKN3XPKDE6A, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.477696651 (592h36m49.348779696s) [noise=true , address=45.91.116.9:4262 , piece_id=U4PLHTD3NAVJO5HFG4ESUXNOCGQGZFVTMIRRWS3Y6EF2OQLMSWDA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.477860333 (592h36m49.348943368s) [noise=true , address=145.79.227.18:4006 , piece_id=CTMWS2HVHHSN5Z6TRJU42V35GU664OCFE4S2TYFIGCAZBNH4I2SA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.478663782 (592h36m49.349746827s) [noise=true , address=145.79.227.16:4912 , piece_id=UKNWWNL34SA7SBRNTHFHJSFXZ25Y6HAXOAOLJZK6U6SHKAMGRB2A, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.478803088 (592h36m49.349886213s) [noise=true , address=45.91.116.12:4226 , piece_id=T4Q5H657AXB2FOD3G2HOVNBFSGTJK4A4HNVYNCEYQQLPX53EMCOQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.485394589 (592h36m49.356477634s) [noise=true , address=45.91.116.8:4046 , piece_id=OJPSDMAY26O64KW74R2DB6PFRQISHUTZZMRZP5DREGWVK4N3I4YQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:51.189904377 (592h36m51.060987412s) [upload=519528]: Next returned nil segment Dec 30 08:05:51.189931058 (592h36m51.061014103s) [upload=519528]: Waiting for error group managing segments... Dec 30 08:05:51.478112981 (592h36m51.349196016s) [noise=true , address=145.79.227.4:4187 , piece_id=JK2NEA6QVPEEEAKIPLTQZDLCQIIW4YJ7UN7HN55OPPSAB6TAUO5A, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.479586027 (592h36m51.350669072s) [noise=true , address=145.79.227.18:4006 , piece_id=CTMWS2HVHHSN5Z6TRJU42V35GU664OCFE4S2TYFIGCAZBNH4I2SA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.485791633 (592h36m51.356874668s) [noise=true , address=145.79.227.16:4912 , piece_id=UKNWWNL34SA7SBRNTHFHJSFXZ25Y6HAXOAOLJZK6U6SHKAMGRB2A, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.492754882 (592h36m51.363837927s) [noise=true , address=145.79.227.20:4438 , piece_id=7G5AQ2RA4S4VWCSWVMKJYZP7NRVFGSYBMZXIB37QDXPFJYH44OWA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.499096197 (592h36m51.370179272s) [noise=true , address=145.79.227.36:4519 , piece_id=NJ2AMF5VCXB2LK2ADGUQTOV74VOL7TWIVTZKUQZUJIENECJIDXWA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.555063718 (592h36m51.426146763s) [noise=true , address=145.79.227.3:4150 , piece_id=MFGBL7CDWLDFR5RNIX36E64VRKAOFLZEMHLKUMDH6AKN3XPKDE6A, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.556999093 (592h36m51.428082138s) [noise=true , address=145.79.227.10:4401 , piece_id=R752IMJI53EQ2L3B7WJY3FSWVPTHMRS4DHP6HH572F3E6I5JQQKQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.670387863 (592h36m51.541470898s) [noise=true , address=145.79.227.40:4051 , piece_id=RBKROK3UFJ27GFE23EFUY5557W2OOK2MN5DJZQFUUFNF7DFLUESA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.829912243 (592h36m51.700995288s) [noise=true , address=45.91.116.12:4226 , piece_id=T4Q5H657AXB2FOD3G2HOVNBFSGTJK4A4HNVYNCEYQQLPX53EMCOQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.833577082 (592h36m51.704660117s) [noise=true , address=45.91.116.5:4300 , piece_id=NVGIUH7A7KSFLOGXHTYR3MS4EN6QFGFVE727BP52YW537ZOWJNRQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.903958540 (592h36m51.775041575s) [noise=true , address=45.91.116.8:4046 , piece_id=OJPSDMAY26O64KW74R2DB6PFRQISHUTZZMRZP5DREGWVK4N3I4YQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.966542790 (592h36m51.837625835s) [noise=true , address=45.91.116.11:4120 , piece_id=3D4EBYSGHWBRWOTLVIBLHPCSZSAZ7H2HBE2256HFHVIRZVKWUYTA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:52.369380267 (592h36m52.240463312s) [noise=true , address=77.90.31.21:4880 , piece_id=HWTZMJVYTPYAYFZ4R35LOY4BZ7EIDAK7FIPSQKNC7R4IBQKX25LA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:52.369410324 (592h36m52.240493359s) [seg_pos={0 0}, upload=519528]: Segment reached optimal threshold of 13 pieces. Dec 30 08:05:52.369686961 (592h36m52.240769996s) [noise=true , address=45.91.116.6:4976 , piece_id=VQAPJJXFX3F5FKDAYDOQJLO2JCULT2JM4N6NEHBJ4GEYVXDYOARQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:60552->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:60552->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60552->45.91.116.6:4976: use of closed network connection Dec 30 08:05:52.369696248 (592h36m52.240779293s) [noise=true , address=45.91.116.6:4976 , piece_id=VQAPJJXFX3F5FKDAYDOQJLO2JCULT2JM4N6NEHBJ4GEYVXDYOARQ, seg_pos={0 0}, upload=519528]: Long tail cancellation due to optimal threshold Dec 30 08:05:52.369817389 (592h36m52.240900434s) [noise=true , address=45.91.116.9:4262 , piece_id=U4PLHTD3NAVJO5HFG4ESUXNOCGQGZFVTMIRRWS3Y6EF2OQLMSWDA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 08:05:52.369838730 (592h36m52.240921775s) [noise=true , address=45.91.116.9:4262 , piece_id=U4PLHTD3NAVJO5HFG4ESUXNOCGQGZFVTMIRRWS3Y6EF2OQLMSWDA, seg_pos={0 0}, upload=519528]: Long tail cancellation due to optimal threshold Dec 30 08:05:52.370009976 (592h36m52.241093011s) [noise=true , address=77.90.31.19:4844 , piece_id=QS3RPY5XUJIBA4DTBGRJ2KPTNYH7V2UZK3NEG76YBRBWCVB7LS3A, seg_pos={0 0}, upload=519528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 08:05:52.370029663 (592h36m52.241112708s) [noise=true , address=77.90.31.19:4844 , piece_id=QS3RPY5XUJIBA4DTBGRJ2KPTNYH7V2UZK3NEG76YBRBWCVB7LS3A, seg_pos={0 0}, upload=519528]: Long tail cancellation due to optimal threshold Dec 30 08:05:52.370055893 (592h36m52.241138938s) [seg_pos={0 0}, upload=519528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:05:52.370160051 (592h36m52.241243096s) [upload=519528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:05:52.370165782 (592h36m52.241248827s) [upload=519528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:05:52.666539645 (592h36m52.53762268s) [upload=519528]: Done uploading segments... Dec 30 08:05:52.666597045 (592h36m52.53768009s) [upload=519528]: Upload finished. err: <nil> Duration: 5.97456115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.....X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X......] duration: 3.461145101s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:06:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461221948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX....X....X.XX..X....XX.XX..................X...X..X..X..XX] duration: 30.006355666s trace: https://jaeger.dev.storj.io/trace/71628c77d07a1aac FAIL (2025-12-30 08:06:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: T4Q5H657AXB2FOD3G2HOVNBFSGTJK4A4HNVYNCEYQQLPX53EMCOQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: OJPSDMAY26O64KW74R2DB6PFRQISHUTZZMRZP5DREGWVK4N3I4YQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NVGIUH7A7KSFLOGXHTYR3MS4EN6QFGFVE727BP52YW537ZOWJNRQ) context deadline exceeded ======LOGS====== 0 08:05:49.227438369 (592h36m49.098521414s) [upload=519528]: Starting upload Dec 30 08:05:49.227549210 (592h36m49.098632275s) [upload=519528]: Done starting upload Dec 30 08:05:49.227611589 (592h36m49.098694634s) [upload=519528]: Uploading segments... Dec 30 08:05:49.227925045 (592h36m49.099008091s) [upload=519528]: Got next segment. Inline: false Dec 30 08:05:49.227986202 (592h36m49.099069247s) [upload=519528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:05:49.227993105 (592h36m49.09907615s) [upload=519528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:05:49.474216123 (592h36m49.345299158s) [seg_pos={0 0}, upload=519528]: Begin upload segment... Dec 30 08:05:49.474271188 (592h36m49.345354223s) [seg_pos={0 0}, upload=519528]: Done begin upload segment. Dec 30 08:05:49.474337104 (592h36m49.345420149s) [noise=true , address=45.91.116.11:4120 , piece_id=3D4EBYSGHWBRWOTLVIBLHPCSZSAZ7H2HBE2256HFHVIRZVKWUYTA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.474399472 (592h36m49.345482507s) [noise=true , address=145.79.227.20:4438 , piece_id=7G5AQ2RA4S4VWCSWVMKJYZP7NRVFGSYBMZXIB37QDXPFJYH44OWA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.475006188 (592h36m49.346089223s) [noise=true , address=145.79.227.36:4519 , piece_id=NJ2AMF5VCXB2LK2ADGUQTOV74VOL7TWIVTZKUQZUJIENECJIDXWA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.475438661 (592h36m49.346521706s) [noise=true , address=145.79.227.40:4051 , piece_id=RBKROK3UFJ27GFE23EFUY5557W2OOK2MN5DJZQFUUFNF7DFLUESA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.475915770 (592h36m49.346998815s) [noise=true , address=145.79.227.10:4401 , piece_id=R752IMJI53EQ2L3B7WJY3FSWVPTHMRS4DHP6HH572F3E6I5JQQKQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.475926460 (592h36m49.347009505s) [noise=true , address=45.91.116.5:4300 , piece_id=NVGIUH7A7KSFLOGXHTYR3MS4EN6QFGFVE727BP52YW537ZOWJNRQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.476342642 (592h36m49.347425677s) [noise=true , address=77.90.31.19:4844 , piece_id=QS3RPY5XUJIBA4DTBGRJ2KPTNYH7V2UZK3NEG76YBRBWCVB7LS3A, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.476383560 (592h36m49.347466595s) [noise=true , address=77.90.31.21:4880 , piece_id=HWTZMJVYTPYAYFZ4R35LOY4BZ7EIDAK7FIPSQKNC7R4IBQKX25LA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.476799392 (592h36m49.347882427s) [noise=true , address=45.91.116.6:4976 , piece_id=VQAPJJXFX3F5FKDAYDOQJLO2JCULT2JM4N6NEHBJ4GEYVXDYOARQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.477105054 (592h36m49.348188089s) [noise=true , address=145.79.227.4:4187 , piece_id=JK2NEA6QVPEEEAKIPLTQZDLCQIIW4YJ7UN7HN55OPPSAB6TAUO5A, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.477403463 (592h36m49.348486498s) [noise=true , address=145.79.227.3:4150 , piece_id=MFGBL7CDWLDFR5RNIX36E64VRKAOFLZEMHLKUMDH6AKN3XPKDE6A, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.477696651 (592h36m49.348779696s) [noise=true , address=45.91.116.9:4262 , piece_id=U4PLHTD3NAVJO5HFG4ESUXNOCGQGZFVTMIRRWS3Y6EF2OQLMSWDA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.477860333 (592h36m49.348943368s) [noise=true , address=145.79.227.18:4006 , piece_id=CTMWS2HVHHSN5Z6TRJU42V35GU664OCFE4S2TYFIGCAZBNH4I2SA, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.478663782 (592h36m49.349746827s) [noise=true , address=145.79.227.16:4912 , piece_id=UKNWWNL34SA7SBRNTHFHJSFXZ25Y6HAXOAOLJZK6U6SHKAMGRB2A, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.478803088 (592h36m49.349886213s) [noise=true , address=45.91.116.12:4226 , piece_id=T4Q5H657AXB2FOD3G2HOVNBFSGTJK4A4HNVYNCEYQQLPX53EMCOQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:49.485394589 (592h36m49.356477634s) [noise=true , address=45.91.116.8:4046 , piece_id=OJPSDMAY26O64KW74R2DB6PFRQISHUTZZMRZP5DREGWVK4N3I4YQ, seg_pos={0 0}, upload=519528]: Uploading piece... Dec 30 08:05:51.189904377 (592h36m51.060987412s) [upload=519528]: Next returned nil segment Dec 30 08:05:51.189931058 (592h36m51.061014103s) [upload=519528]: Waiting for error group managing segments... Dec 30 08:05:51.478112981 (592h36m51.349196016s) [noise=true , address=145.79.227.4:4187 , piece_id=JK2NEA6QVPEEEAKIPLTQZDLCQIIW4YJ7UN7HN55OPPSAB6TAUO5A, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.479586027 (592h36m51.350669072s) [noise=true , address=145.79.227.18:4006 , piece_id=CTMWS2HVHHSN5Z6TRJU42V35GU664OCFE4S2TYFIGCAZBNH4I2SA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.485791633 (592h36m51.356874668s) [noise=true , address=145.79.227.16:4912 , piece_id=UKNWWNL34SA7SBRNTHFHJSFXZ25Y6HAXOAOLJZK6U6SHKAMGRB2A, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.492754882 (592h36m51.363837927s) [noise=true , address=145.79.227.20:4438 , piece_id=7G5AQ2RA4S4VWCSWVMKJYZP7NRVFGSYBMZXIB37QDXPFJYH44OWA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.499096197 (592h36m51.370179272s) [noise=true , address=145.79.227.36:4519 , piece_id=NJ2AMF5VCXB2LK2ADGUQTOV74VOL7TWIVTZKUQZUJIENECJIDXWA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.555063718 (592h36m51.426146763s) [noise=true , address=145.79.227.3:4150 , piece_id=MFGBL7CDWLDFR5RNIX36E64VRKAOFLZEMHLKUMDH6AKN3XPKDE6A, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.556999093 (592h36m51.428082138s) [noise=true , address=145.79.227.10:4401 , piece_id=R752IMJI53EQ2L3B7WJY3FSWVPTHMRS4DHP6HH572F3E6I5JQQKQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.670387863 (592h36m51.541470898s) [noise=true , address=145.79.227.40:4051 , piece_id=RBKROK3UFJ27GFE23EFUY5557W2OOK2MN5DJZQFUUFNF7DFLUESA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.829912243 (592h36m51.700995288s) [noise=true , address=45.91.116.12:4226 , piece_id=T4Q5H657AXB2FOD3G2HOVNBFSGTJK4A4HNVYNCEYQQLPX53EMCOQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.833577082 (592h36m51.704660117s) [noise=true , address=45.91.116.5:4300 , piece_id=NVGIUH7A7KSFLOGXHTYR3MS4EN6QFGFVE727BP52YW537ZOWJNRQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.903958540 (592h36m51.775041575s) [noise=true , address=45.91.116.8:4046 , piece_id=OJPSDMAY26O64KW74R2DB6PFRQISHUTZZMRZP5DREGWVK4N3I4YQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:51.966542790 (592h36m51.837625835s) [noise=true , address=45.91.116.11:4120 , piece_id=3D4EBYSGHWBRWOTLVIBLHPCSZSAZ7H2HBE2256HFHVIRZVKWUYTA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:52.369380267 (592h36m52.240463312s) [noise=true , address=77.90.31.21:4880 , piece_id=HWTZMJVYTPYAYFZ4R35LOY4BZ7EIDAK7FIPSQKNC7R4IBQKX25LA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: <nil> Dec 30 08:05:52.369410324 (592h36m52.240493359s) [seg_pos={0 0}, upload=519528]: Segment reached optimal threshold of 13 pieces. Dec 30 08:05:52.369686961 (592h36m52.240769996s) [noise=true , address=45.91.116.6:4976 , piece_id=VQAPJJXFX3F5FKDAYDOQJLO2JCULT2JM4N6NEHBJ4GEYVXDYOARQ, seg_pos={0 0}, upload=519528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:60552->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:60552->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60552->45.91.116.6:4976: use of closed network connection Dec 30 08:05:52.369696248 (592h36m52.240779293s) [noise=true , address=45.91.116.6:4976 , piece_id=VQAPJJXFX3F5FKDAYDOQJLO2JCULT2JM4N6NEHBJ4GEYVXDYOARQ, seg_pos={0 0}, upload=519528]: Long tail cancellation due to optimal threshold Dec 30 08:05:52.369817389 (592h36m52.240900434s) [noise=true , address=45.91.116.9:4262 , piece_id=U4PLHTD3NAVJO5HFG4ESUXNOCGQGZFVTMIRRWS3Y6EF2OQLMSWDA, seg_pos={0 0}, upload=519528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 08:05:52.369838730 (592h36m52.240921775s) [noise=true , address=45.91.116.9:4262 , piece_id=U4PLHTD3NAVJO5HFG4ESUXNOCGQGZFVTMIRRWS3Y6EF2OQLMSWDA, seg_pos={0 0}, upload=519528]: Long tail cancellation due to optimal threshold Dec 30 08:05:52.370009976 (592h36m52.241093011s) [noise=true , address=77.90.31.19:4844 , piece_id=QS3RPY5XUJIBA4DTBGRJ2KPTNYH7V2UZK3NEG76YBRBWCVB7LS3A, seg_pos={0 0}, upload=519528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 08:05:52.370029663 (592h36m52.241112708s) [noise=true , address=77.90.31.19:4844 , piece_id=QS3RPY5XUJIBA4DTBGRJ2KPTNYH7V2UZK3NEG76YBRBWCVB7LS3A, seg_pos={0 0}, upload=519528]: Long tail cancellation due to optimal threshold Dec 30 08:05:52.370055893 (592h36m52.241138938s) [seg_pos={0 0}, upload=519528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:05:52.370160051 (592h36m52.241243096s) [upload=519528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:05:52.370165782 (592h36m52.241248827s) [upload=519528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:05:52.666539645 (592h36m52.53762268s) [upload=519528]: Done uploading segments... Dec 30 08:05:52.666597045 (592h36m52.53768009s) [upload=519528]: Upload finished. err: <nil> Duration: 30.006377506s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [......X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X.....] duration: 3.615433801s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:06:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.615446725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 42.663541831s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:05:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 42.66357355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX....X....X.XX..X....XX.XX..................X...X..X..X..X] duration: 30.002139229s trace: https://jaeger.dev.storj.io/trace/490aa8ac70aa5fa FAIL (2025-12-30 08:05:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 7623UJGOM52TKCJH4GDE24H3RI22B5VMI2NSXFD3B7XUEEQIL4RA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: LD5TIJT5CPMHXD7TM5SQYXGJXRNR3SHZSE4GCYAKQQOMJDOFJ3FA) context deadline exceeded ======LOGS====== Dec 30 08:05:15.971274329 (592h36m15.842357375s) [upload=519518]: Starting upload Dec 30 08:05:15.971373108 (592h36m15.842456143s) [upload=519518]: Done starting upload Dec 30 08:05:15.971480642 (592h36m15.842563718s) [upload=519518]: Uploading segments... Dec 30 08:05:15.971619777 (592h36m15.842702812s) [upload=519518]: Got next segment. Inline: false Dec 30 08:05:15.971667838 (592h36m15.842750884s) [upload=519518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:05:15.971672408 (592h36m15.842755443s) [upload=519518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:05:16.219200249 (592h36m16.090283294s) [seg_pos={0 0}, upload=519518]: Begin upload segment... Dec 30 08:05:16.219364823 (592h36m16.090447858s) [seg_pos={0 0}, upload=519518]: Done begin upload segment. Dec 30 08:05:16.219442080 (592h36m16.090525115s) [noise=true , address=45.91.116.11:4117 , piece_id=ZQ3CCYMFSPCL2G663HQCWPHWEEVGGMGZER772PVKJQPH6RQKB5FA, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.220341372 (592h36m16.091424407s) [noise=true , address=145.79.227.11:4732 , piece_id=RUEQTQCNTFHU6CW65DQ5IMP4VMO7QEMFACKAQOYTZ2MBA2EH53QA, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.221018371 (592h36m16.092101416s) [noise=true , address=145.79.227.51:4411 , piece_id=G7FDZEQJJP4XNDTG53C2Z3CP4V2FRRBINVQZSDQYGFYVATDAKHZQ, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.221423052 (592h36m16.092506107s) [noise=true , address=145.79.227.19:4079 , piece_id=GIAPNWS7J7P6ADTQORHYF5YJS74KRFQESI26EJCL66D5R3NAO2DQ, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.222074082 (592h36m16.093157127s) [noise=true , address=77.90.31.13:4627 , piece_id=LD5TIJT5CPMHXD7TM5SQYXGJXRNR3SHZSE4GCYAKQQOMJDOFJ3FA, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.222924581 (592h36m16.094007636s) [noise=true , address=145.79.227.31:4798 , piece_id=D7LFSQDUDVUJBVS4XRP6HCVMASCEP4KFCEETBGSV5IY4PWH2YFRQ, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.223437578 (592h36m16.094520653s) [noise=true , address=77.90.31.19:4841 , piece_id=MUB756NKM2ICYIKI7XKF4OHC3AT5BYO6HXPBNFRNJBYLXQDGUMHA, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.223848551 (592h36m16.094931586s) [noise=true , address=145.79.227.39:4013 , piece_id=4ELQFKDPGLFKVZRHUWU6YHIWD4BCROHUCVVW3BUXISAX5DVXQ2UQ, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.224086213 (592h36m16.095169249s) [noise=true , address=145.79.227.20:4439 , piece_id=U2MHETDGBWH7HHI4K5Z6KILCV2E5QKT5NZLWA7RHV2ZMGPVX3JHA, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.224391334 (592h36m16.095474379s) [noise=true , address=145.79.227.12:4765 , piece_id=43PRWLIBCCAIOE2DPBLQUVDJDR2CM4YPAAG4ZUGWGKNSJOWRUB3A, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.224453482 (592h36m16.095536527s) [noise=true , address=145.79.227.38:4979 , piece_id=YSOVLWKHNETYDO5Z7XM5IZAEIX4HQIH24P4Y4GOZZCN3PKGYDFVQ, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.224892649 (592h36m16.095975694s) [noise=true , address=77.90.31.17:4773 , piece_id=ROYESAKYKRHPTJBFMQERUJHEQKEXMBSXCMFUPDWGTHJLXMCOAQHQ, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.225032665 (592h36m16.096115711s) [noise=true , address=77.90.31.18:4808 , piece_id=7623UJGOM52TKCJH4GDE24H3RI22B5VMI2NSXFD3B7XUEEQIL4RA, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.225579216 (592h36m16.096662261s) [noise=true , address=45.91.116.7:4012 , piece_id=K3RH6BTKD426BM5RQSB2CMVVXZ6PMXMJQGMYTKEMQ2ODZMATSL2Q, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.229272760 (592h36m16.100355805s) [noise=true , address=145.79.227.10:4403 , piece_id=2K7VI6IDQJH3ZFSBCH6SCKCYLUJFUB35TPI6Z4CLJP2DY5GF4LJQ, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:16.230463126 (592h36m16.101546161s) [noise=true , address=145.79.227.47:4266 , piece_id=NJ5XRFMASLL4JCFXWUGJLZKUR7EEYMF7ROOKBTQU5UGAXMIQOR6Q, seg_pos={0 0}, upload=519518]: Uploading piece... Dec 30 08:05:17.931366346 (592h36m17.802449391s) [upload=519518]: Next returned nil segment Dec 30 08:05:17.931386605 (592h36m17.80246965s) [upload=519518]: Waiting for error group managing segments... Dec 30 08:05:18.213827207 (592h36m18.084910242s) [noise=true , address=145.79.227.19:4079 , piece_id=GIAPNWS7J7P6ADTQORHYF5YJS74KRFQESI26EJCL66D5R3NAO2DQ, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.214353870 (592h36m18.085436905s) [noise=true , address=145.79.227.11:4732 , piece_id=RUEQTQCNTFHU6CW65DQ5IMP4VMO7QEMFACKAQOYTZ2MBA2EH53QA, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.215308207 (592h36m18.086391242s) [noise=true , address=145.79.227.51:4411 , piece_id=G7FDZEQJJP4XNDTG53C2Z3CP4V2FRRBINVQZSDQYGFYVATDAKHZQ, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.220335950 (592h36m18.091418995s) [noise=true , address=145.79.227.10:4403 , piece_id=2K7VI6IDQJH3ZFSBCH6SCKCYLUJFUB35TPI6Z4CLJP2DY5GF4LJQ, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.233948420 (592h36m18.105031465s) [noise=true , address=145.79.227.47:4266 , piece_id=NJ5XRFMASLL4JCFXWUGJLZKUR7EEYMF7ROOKBTQU5UGAXMIQOR6Q, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.236639495 (592h36m18.10772254s) [noise=true , address=145.79.227.31:4798 , piece_id=D7LFSQDUDVUJBVS4XRP6HCVMASCEP4KFCEETBGSV5IY4PWH2YFRQ, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.243742150 (592h36m18.114825195s) [noise=true , address=145.79.227.38:4979 , piece_id=YSOVLWKHNETYDO5Z7XM5IZAEIX4HQIH24P4Y4GOZZCN3PKGYDFVQ, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.297528157 (592h36m18.168611192s) [noise=true , address=145.79.227.20:4439 , piece_id=U2MHETDGBWH7HHI4K5Z6KILCV2E5QKT5NZLWA7RHV2ZMGPVX3JHA, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.301046717 (592h36m18.172129762s) [noise=true , address=145.79.227.39:4013 , piece_id=4ELQFKDPGLFKVZRHUWU6YHIWD4BCROHUCVVW3BUXISAX5DVXQ2UQ, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.302368493 (592h36m18.173451539s) [noise=true , address=145.79.227.12:4765 , piece_id=43PRWLIBCCAIOE2DPBLQUVDJDR2CM4YPAAG4ZUGWGKNSJOWRUB3A, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.398585746 (592h36m18.269668801s) [noise=true , address=77.90.31.13:4627 , piece_id=LD5TIJT5CPMHXD7TM5SQYXGJXRNR3SHZSE4GCYAKQQOMJDOFJ3FA, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.517840576 (592h36m18.388923631s) [noise=true , address=77.90.31.18:4808 , piece_id=7623UJGOM52TKCJH4GDE24H3RI22B5VMI2NSXFD3B7XUEEQIL4RA, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.526058764 (592h36m18.397141799s) [noise=true , address=45.91.116.11:4117 , piece_id=ZQ3CCYMFSPCL2G663HQCWPHWEEVGGMGZER772PVKJQPH6RQKB5FA, seg_pos={0 0}, upload=519518]: Done uploading piece. err: <nil> Dec 30 08:05:18.526085906 (592h36m18.397168961s) [seg_pos={0 0}, upload=519518]: Segment reached optimal threshold of 13 pieces. Dec 30 08:05:18.526372832 (592h36m18.397455878s) [noise=true , address=77.90.31.19:4841 , piece_id=MUB756NKM2ICYIKI7XKF4OHC3AT5BYO6HXPBNFRNJBYLXQDGUMHA, seg_pos={0 0}, upload=519518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 08:05:18.526382732 (592h36m18.397465767s) [noise=true , address=77.90.31.19:4841 , piece_id=MUB756NKM2ICYIKI7XKF4OHC3AT5BYO6HXPBNFRNJBYLXQDGUMHA, seg_pos={0 0}, upload=519518]: Long tail cancellation due to optimal threshold Dec 30 08:05:18.527139302 (592h36m18.398222367s) [noise=true , address=77.90.31.17:4773 , piece_id=ROYESAKYKRHPTJBFMQERUJHEQKEXMBSXCMFUPDWGTHJLXMCOAQHQ, seg_pos={0 0}, upload=519518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 30 08:05:18.527164781 (592h36m18.398247816s) [noise=true , address=77.90.31.17:4773 , piece_id=ROYESAKYKRHPTJBFMQERUJHEQKEXMBSXCMFUPDWGTHJLXMCOAQHQ, seg_pos={0 0}, upload=519518]: Long tail cancellation due to optimal threshold Dec 30 08:05:18.527206239 (592h36m18.398289325s) [noise=true , address=45.91.116.7:4012 , piece_id=K3RH6BTKD426BM5RQSB2CMVVXZ6PMXMJQGMYTKEMQ2ODZMATSL2Q, seg_pos={0 0}, upload=519518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: read tcp 188.166.225.170:49832->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 08:05:18.527220167 (592h36m18.398303212s) [noise=true , address=45.91.116.7:4012 , piece_id=K3RH6BTKD426BM5RQSB2CMVVXZ6PMXMJQGMYTKEMQ2ODZMATSL2Q, seg_pos={0 0}, upload=519518]: Long tail cancellation due to optimal threshold Dec 30 08:05:18.527248570 (592h36m18.398331605s) [seg_pos={0 0}, upload=519518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:05:18.527421118 (592h36m18.398504164s) [upload=519518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:05:18.527430637 (592h36m18.398513682s) [upload=519518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:05:18.828085653 (592h36m18.699168688s) [upload=519518]: Done uploading segments... Dec 30 08:05:18.828110319 (592h36m18.699193364s) [upload=519518]: Upload finished. err: <nil> Duration: 30.002163485s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.......X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X....] duration: 3.354420299s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:05:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.354450807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 15.109541119s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:04:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 15.10956219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XXX....X....X.XX..X....XX.XX..................X...X..X..X..] duration: 5.889681456s trace: https://jaeger.dev.storj.io/trace/174b4a72dc276c8e FAIL (2025-12-30 08:04:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: 5O7ZYTMX52MMTXLKNWYMNYDDIGXMAPIL77TJSZJAUBFQHZ4MDVFA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: RHPNY6DGOFKWIIUWFPAVYUPGD5SINESJ7AUI5YJJTPUTBKWTD4OQ) context deadline exceeded ======LOGS====== Dec 30 08:03:09.698527656 (592h34m9.569610691s) [upload=519486]: Starting upload Dec 30 08:03:09.698582189 (592h34m9.569665235s) [upload=519486]: Done starting upload Dec 30 08:03:09.698755720 (592h34m9.569838755s) [upload=519486]: Uploading segments... Dec 30 08:03:09.698788833 (592h34m9.569871868s) [upload=519486]: Got next segment. Inline: false Dec 30 08:03:09.698813390 (592h34m9.569896435s) [upload=519486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:03:09.698817608 (592h34m9.569900653s) [upload=519486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:03:09.951847797 (592h34m9.822930832s) [seg_pos={0 0}, upload=519486]: Begin upload segment... Dec 30 08:03:09.951891340 (592h34m9.822974375s) [seg_pos={0 0}, upload=519486]: Done begin upload segment. Dec 30 08:03:09.951917099 (592h34m9.823000134s) [noise=true , address=45.91.116.4:4940 , piece_id=FR5NT7NITJEDQUV5WWI7XU4IRYZ77S5OW4N4Y2HCZLH3UFSVJ4CQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.956278684 (592h34m9.827361719s) [noise=true , address=145.79.227.3:4151 , piece_id=WCNZYT5AGKC4VUESFZJMBOZXD5DXZOZBD55G6I3YRZCKU7DWZSBQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.957651439 (592h34m9.828734474s) [noise=true , address=45.91.116.14:4189 , piece_id=N6UOCLGM4ESTGLM5VO35RFZIIUX7MJVDOAPNZOJICZIER3CN4HKQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.958991560 (592h34m9.830074605s) [noise=true , address=145.79.227.22:4510 , piece_id=DVLQGVMACOGOPJBZ6ILHWH2QL3RA73T567EAXOF5UOVXAA5X2NJA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.960336431 (592h34m9.831419466s) [noise=true , address=145.79.227.23:4546 , piece_id=6LFZ7IQQREVOTAJ5ZV6X6E7SUEXGFFKR2Q6AS62GE7QANHIFBWDA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.961615577 (592h34m9.832698622s) [noise=true , address=145.79.227.16:4910 , piece_id=Q45RRY3XPAZV6F6BQB7FLXK4GFU7RX3EJPA3VA2S6PXK4L5TKN6A, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.963400195 (592h34m9.83448324s) [noise=true , address=145.79.227.32:4834 , piece_id=NUU2OTVN3KEOWLZ4VQ2R3DOHMTBKPDNLHRXL4TD2QWEZ4MRCJA4A, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.964939095 (592h34m9.836022141s) [noise=true , address=45.91.116.13:4156 , piece_id=A252VG3RXAOI3FNNROHPVU4GM7W7A224VG3Z5R5BPBWBPC6J42IA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.966446816 (592h34m9.837529861s) [noise=true , address=145.79.227.13:4802 , piece_id=C44TZGSPG47MHDKTOMTZUBHTTYH2XWQ3GDIHV2PBA2NLE3V4YM7A, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.968154818 (592h34m9.839237853s) [noise=true , address=145.79.227.11:4732 , piece_id=XHAGS7SZSUSXSMPG25J2ZSSF5ALU7ACMWOJDA37V2W2ES2FLU5RQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.969579672 (592h34m9.840662707s) [noise=true , address=77.90.31.17:4773 , piece_id=LSYUMJ4EHV5Y74UABQCIPKOIIGN25AH2463VBQVMMYX5W37N5JVQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.971111428 (592h34m9.842194473s) [noise=true , address=77.90.31.14:4661 , piece_id=5O7ZYTMX52MMTXLKNWYMNYDDIGXMAPIL77TJSZJAUBFQHZ4MDVFA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.974254504 (592h34m9.845337549s) [noise=true , address=145.79.227.37:4566 , piece_id=RBN26VQGCFCWFMD3LJK6JXLYIVDUUCUROZA3ZFNFKBPNMDEQTC7Q, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.976059310 (592h34m9.847142355s) [noise=true , address=77.90.31.11:4555 , piece_id=RHPNY6DGOFKWIIUWFPAVYUPGD5SINESJ7AUI5YJJTPUTBKWTD4OQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.977694343 (592h34m9.848777378s) [noise=true , address=45.91.116.6:4975 , piece_id=2QG26HVQS3O4B4QZJ3ASJD27OG2V3EMEAHDSWJRJIGOSQV2VWF2Q, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.978997956 (592h34m9.850081001s) [noise=true , address=145.79.227.39:4014 , piece_id=2V2OJTOX7QNBZP6FJDVGN2UIYECGDNV4JHBNN4TTJCIPBZN5PTPQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:11.704904506 (592h34m11.575987541s) [upload=519486]: Next returned nil segment Dec 30 08:03:11.704920005 (592h34m11.576003051s) [upload=519486]: Waiting for error group managing segments... Dec 30 08:03:11.989912006 (592h34m11.860995041s) [noise=true , address=145.79.227.13:4802 , piece_id=C44TZGSPG47MHDKTOMTZUBHTTYH2XWQ3GDIHV2PBA2NLE3V4YM7A, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:11.995741767 (592h34m11.866824802s) [noise=true , address=145.79.227.3:4151 , piece_id=WCNZYT5AGKC4VUESFZJMBOZXD5DXZOZBD55G6I3YRZCKU7DWZSBQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:11.996367138 (592h34m11.867450173s) [noise=true , address=145.79.227.39:4014 , piece_id=2V2OJTOX7QNBZP6FJDVGN2UIYECGDNV4JHBNN4TTJCIPBZN5PTPQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.001786587 (592h34m11.872869622s) [noise=true , address=145.79.227.22:4510 , piece_id=DVLQGVMACOGOPJBZ6ILHWH2QL3RA73T567EAXOF5UOVXAA5X2NJA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.017945095 (592h34m11.88902813s) [noise=true , address=145.79.227.16:4910 , piece_id=Q45RRY3XPAZV6F6BQB7FLXK4GFU7RX3EJPA3VA2S6PXK4L5TKN6A, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.022459421 (592h34m11.893542467s) [noise=true , address=145.79.227.32:4834 , piece_id=NUU2OTVN3KEOWLZ4VQ2R3DOHMTBKPDNLHRXL4TD2QWEZ4MRCJA4A, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.026947017 (592h34m11.898030052s) [noise=true , address=145.79.227.37:4566 , piece_id=RBN26VQGCFCWFMD3LJK6JXLYIVDUUCUROZA3ZFNFKBPNMDEQTC7Q, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.073022111 (592h34m11.944105156s) [noise=true , address=145.79.227.11:4732 , piece_id=XHAGS7SZSUSXSMPG25J2ZSSF5ALU7ACMWOJDA37V2W2ES2FLU5RQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.073930170 (592h34m11.945013215s) [noise=true , address=145.79.227.23:4546 , piece_id=6LFZ7IQQREVOTAJ5ZV6X6E7SUEXGFFKR2Q6AS62GE7QANHIFBWDA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.153017955 (592h34m12.02410103s) [noise=true , address=77.90.31.11:4555 , piece_id=RHPNY6DGOFKWIIUWFPAVYUPGD5SINESJ7AUI5YJJTPUTBKWTD4OQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.233887934 (592h34m12.104970979s) [noise=true , address=77.90.31.17:4773 , piece_id=LSYUMJ4EHV5Y74UABQCIPKOIIGN25AH2463VBQVMMYX5W37N5JVQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.285404190 (592h34m12.156487235s) [noise=true , address=77.90.31.14:4661 , piece_id=5O7ZYTMX52MMTXLKNWYMNYDDIGXMAPIL77TJSZJAUBFQHZ4MDVFA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.317923983 (592h34m12.189007018s) [noise=true , address=45.91.116.6:4975 , piece_id=2QG26HVQS3O4B4QZJ3ASJD27OG2V3EMEAHDSWJRJIGOSQV2VWF2Q, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.317950723 (592h34m12.189033758s) [seg_pos={0 0}, upload=519486]: Segment reached optimal threshold of 13 pieces. Dec 30 08:03:12.318216570 (592h34m12.189299615s) [noise=true , address=45.91.116.14:4189 , piece_id=N6UOCLGM4ESTGLM5VO35RFZIIUX7MJVDOAPNZOJICZIER3CN4HKQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:55238->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:55238->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55238->45.91.116.14:4189: use of closed network connection Dec 30 08:03:12.318227741 (592h34m12.189310776s) [noise=true , address=45.91.116.14:4189 , piece_id=N6UOCLGM4ESTGLM5VO35RFZIIUX7MJVDOAPNZOJICZIER3CN4HKQ, seg_pos={0 0}, upload=519486]: Long tail cancellation due to optimal threshold Dec 30 08:03:12.318469020 (592h34m12.189552056s) [noise=true , address=45.91.116.4:4940 , piece_id=FR5NT7NITJEDQUV5WWI7XU4IRYZ77S5OW4N4Y2HCZLH3UFSVJ4CQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 08:03:12.318490882 (592h34m12.189573928s) [noise=true , address=45.91.116.4:4940 , piece_id=FR5NT7NITJEDQUV5WWI7XU4IRYZ77S5OW4N4Y2HCZLH3UFSVJ4CQ, seg_pos={0 0}, upload=519486]: Long tail cancellation due to optimal threshold Dec 30 08:03:12.318563501 (592h34m12.189646536s) [noise=true , address=45.91.116.13:4156 , piece_id=A252VG3RXAOI3FNNROHPVU4GM7W7A224VG3Z5R5BPBWBPC6J42IA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 08:03:12.318573269 (592h34m12.189656314s) [noise=true , address=45.91.116.13:4156 , piece_id=A252VG3RXAOI3FNNROHPVU4GM7W7A224VG3Z5R5BPBWBPC6J42IA, seg_pos={0 0}, upload=519486]: Long tail cancellation due to optimal threshold Dec 30 08:03:12.318635588 (592h34m12.189718633s) [seg_pos={0 0}, upload=519486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:03:12.318744445 (592h34m12.18982749s) [upload=519486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:03:12.318752069 (592h34m12.189835114s) [upload=519486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:03:12.612436200 (592h34m12.483519245s) [upload=519486]: Done uploading segments... Dec 30 08:03:12.612461288 (592h34m12.483544323s) [upload=519486]: Upload finished. err: <nil> Duration: 5.889708476s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 42.971271061s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:04:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 42.971290348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X...] duration: 3.368060662s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:04:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.368088403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XXX....X....X.XX..X....XX.XX..................X...X..X..X.] duration: 5.713477158s trace: https://jaeger.dev.storj.io/trace/174b4a72dc276c8e FAIL (2025-12-30 08:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: 5O7ZYTMX52MMTXLKNWYMNYDDIGXMAPIL77TJSZJAUBFQHZ4MDVFA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: RHPNY6DGOFKWIIUWFPAVYUPGD5SINESJ7AUI5YJJTPUTBKWTD4OQ) context deadline exceeded ======LOGS====== Dec 30 08:03:09.698527656 (592h34m9.569610691s) [upload=519486]: Starting upload Dec 30 08:03:09.698582189 (592h34m9.569665235s) [upload=519486]: Done starting upload Dec 30 08:03:09.698755720 (592h34m9.569838755s) [upload=519486]: Uploading segments... Dec 30 08:03:09.698788833 (592h34m9.569871868s) [upload=519486]: Got next segment. Inline: false Dec 30 08:03:09.698813390 (592h34m9.569896435s) [upload=519486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:03:09.698817608 (592h34m9.569900653s) [upload=519486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:03:09.951847797 (592h34m9.822930832s) [seg_pos={0 0}, upload=519486]: Begin upload segment... Dec 30 08:03:09.951891340 (592h34m9.822974375s) [seg_pos={0 0}, upload=519486]: Done begin upload segment. Dec 30 08:03:09.951917099 (592h34m9.823000134s) [noise=true , address=45.91.116.4:4940 , piece_id=FR5NT7NITJEDQUV5WWI7XU4IRYZ77S5OW4N4Y2HCZLH3UFSVJ4CQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.956278684 (592h34m9.827361719s) [noise=true , address=145.79.227.3:4151 , piece_id=WCNZYT5AGKC4VUESFZJMBOZXD5DXZOZBD55G6I3YRZCKU7DWZSBQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.957651439 (592h34m9.828734474s) [noise=true , address=45.91.116.14:4189 , piece_id=N6UOCLGM4ESTGLM5VO35RFZIIUX7MJVDOAPNZOJICZIER3CN4HKQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.958991560 (592h34m9.830074605s) [noise=true , address=145.79.227.22:4510 , piece_id=DVLQGVMACOGOPJBZ6ILHWH2QL3RA73T567EAXOF5UOVXAA5X2NJA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.960336431 (592h34m9.831419466s) [noise=true , address=145.79.227.23:4546 , piece_id=6LFZ7IQQREVOTAJ5ZV6X6E7SUEXGFFKR2Q6AS62GE7QANHIFBWDA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.961615577 (592h34m9.832698622s) [noise=true , address=145.79.227.16:4910 , piece_id=Q45RRY3XPAZV6F6BQB7FLXK4GFU7RX3EJPA3VA2S6PXK4L5TKN6A, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.963400195 (592h34m9.83448324s) [noise=true , address=145.79.227.32:4834 , piece_id=NUU2OTVN3KEOWLZ4VQ2R3DOHMTBKPDNLHRXL4TD2QWEZ4MRCJA4A, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.964939095 (592h34m9.836022141s) [noise=true , address=45.91.116.13:4156 , piece_id=A252VG3RXAOI3FNNROHPVU4GM7W7A224VG3Z5R5BPBWBPC6J42IA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.966446816 (592h34m9.837529861s) [noise=true , address=145.79.227.13:4802 , piece_id=C44TZGSPG47MHDKTOMTZUBHTTYH2XWQ3GDIHV2PBA2NLE3V4YM7A, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.968154818 (592h34m9.839237853s) [noise=true , address=145.79.227.11:4732 , piece_id=XHAGS7SZSUSXSMPG25J2ZSSF5ALU7ACMWOJDA37V2W2ES2FLU5RQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.969579672 (592h34m9.840662707s) [noise=true , address=77.90.31.17:4773 , piece_id=LSYUMJ4EHV5Y74UABQCIPKOIIGN25AH2463VBQVMMYX5W37N5JVQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.971111428 (592h34m9.842194473s) [noise=true , address=77.90.31.14:4661 , piece_id=5O7ZYTMX52MMTXLKNWYMNYDDIGXMAPIL77TJSZJAUBFQHZ4MDVFA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.974254504 (592h34m9.845337549s) [noise=true , address=145.79.227.37:4566 , piece_id=RBN26VQGCFCWFMD3LJK6JXLYIVDUUCUROZA3ZFNFKBPNMDEQTC7Q, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.976059310 (592h34m9.847142355s) [noise=true , address=77.90.31.11:4555 , piece_id=RHPNY6DGOFKWIIUWFPAVYUPGD5SINESJ7AUI5YJJTPUTBKWTD4OQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.977694343 (592h34m9.848777378s) [noise=true , address=45.91.116.6:4975 , piece_id=2QG26HVQS3O4B4QZJ3ASJD27OG2V3EMEAHDSWJRJIGOSQV2VWF2Q, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.978997956 (592h34m9.850081001s) [noise=true , address=145.79.227.39:4014 , piece_id=2V2OJTOX7QNBZP6FJDVGN2UIYECGDNV4JHBNN4TTJCIPBZN5PTPQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:11.704904506 (592h34m11.575987541s) [upload=519486]: Next returned nil segment Dec 30 08:03:11.704920005 (592h34m11.576003051s) [upload=519486]: Waiting for error group managing segments... Dec 30 08:03:11.989912006 (592h34m11.860995041s) [noise=true , address=145.79.227.13:4802 , piece_id=C44TZGSPG47MHDKTOMTZUBHTTYH2XWQ3GDIHV2PBA2NLE3V4YM7A, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:11.995741767 (592h34m11.866824802s) [noise=true , address=145.79.227.3:4151 , piece_id=WCNZYT5AGKC4VUESFZJMBOZXD5DXZOZBD55G6I3YRZCKU7DWZSBQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:11.996367138 (592h34m11.867450173s) [noise=true , address=145.79.227.39:4014 , piece_id=2V2OJTOX7QNBZP6FJDVGN2UIYECGDNV4JHBNN4TTJCIPBZN5PTPQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.001786587 (592h34m11.872869622s) [noise=true , address=145.79.227.22:4510 , piece_id=DVLQGVMACOGOPJBZ6ILHWH2QL3RA73T567EAXOF5UOVXAA5X2NJA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.017945095 (592h34m11.88902813s) [noise=true , address=145.79.227.16:4910 , piece_id=Q45RRY3XPAZV6F6BQB7FLXK4GFU7RX3EJPA3VA2S6PXK4L5TKN6A, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.022459421 (592h34m11.893542467s) [noise=true , address=145.79.227.32:4834 , piece_id=NUU2OTVN3KEOWLZ4VQ2R3DOHMTBKPDNLHRXL4TD2QWEZ4MRCJA4A, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.026947017 (592h34m11.898030052s) [noise=true , address=145.79.227.37:4566 , piece_id=RBN26VQGCFCWFMD3LJK6JXLYIVDUUCUROZA3ZFNFKBPNMDEQTC7Q, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.073022111 (592h34m11.944105156s) [noise=true , address=145.79.227.11:4732 , piece_id=XHAGS7SZSUSXSMPG25J2ZSSF5ALU7ACMWOJDA37V2W2ES2FLU5RQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.073930170 (592h34m11.945013215s) [noise=true , address=145.79.227.23:4546 , piece_id=6LFZ7IQQREVOTAJ5ZV6X6E7SUEXGFFKR2Q6AS62GE7QANHIFBWDA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.153017955 (592h34m12.02410103s) [noise=true , address=77.90.31.11:4555 , piece_id=RHPNY6DGOFKWIIUWFPAVYUPGD5SINESJ7AUI5YJJTPUTBKWTD4OQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.233887934 (592h34m12.104970979s) [noise=true , address=77.90.31.17:4773 , piece_id=LSYUMJ4EHV5Y74UABQCIPKOIIGN25AH2463VBQVMMYX5W37N5JVQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.285404190 (592h34m12.156487235s) [noise=true , address=77.90.31.14:4661 , piece_id=5O7ZYTMX52MMTXLKNWYMNYDDIGXMAPIL77TJSZJAUBFQHZ4MDVFA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.317923983 (592h34m12.189007018s) [noise=true , address=45.91.116.6:4975 , piece_id=2QG26HVQS3O4B4QZJ3ASJD27OG2V3EMEAHDSWJRJIGOSQV2VWF2Q, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.317950723 (592h34m12.189033758s) [seg_pos={0 0}, upload=519486]: Segment reached optimal threshold of 13 pieces. Dec 30 08:03:12.318216570 (592h34m12.189299615s) [noise=true , address=45.91.116.14:4189 , piece_id=N6UOCLGM4ESTGLM5VO35RFZIIUX7MJVDOAPNZOJICZIER3CN4HKQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:55238->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:55238->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55238->45.91.116.14:4189: use of closed network connection Dec 30 08:03:12.318227741 (592h34m12.189310776s) [noise=true , address=45.91.116.14:4189 , piece_id=N6UOCLGM4ESTGLM5VO35RFZIIUX7MJVDOAPNZOJICZIER3CN4HKQ, seg_pos={0 0}, upload=519486]: Long tail cancellation due to optimal threshold Dec 30 08:03:12.318469020 (592h34m12.189552056s) [noise=true , address=45.91.116.4:4940 , piece_id=FR5NT7NITJEDQUV5WWI7XU4IRYZ77S5OW4N4Y2HCZLH3UFSVJ4CQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 08:03:12.318490882 (592h34m12.189573928s) [noise=true , address=45.91.116.4:4940 , piece_id=FR5NT7NITJEDQUV5WWI7XU4IRYZ77S5OW4N4Y2HCZLH3UFSVJ4CQ, seg_pos={0 0}, upload=519486]: Long tail cancellation due to optimal threshold Dec 30 08:03:12.318563501 (592h34m12.189646536s) [noise=true , address=45.91.116.13:4156 , piece_id=A252VG3RXAOI3FNNROHPVU4GM7W7A224VG3Z5R5BPBWBPC6J42IA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 08:03:12.318573269 (592h34m12.189656314s) [noise=true , address=45.91.116.13:4156 , piece_id=A252VG3RXAOI3FNNROHPVU4GM7W7A224VG3Z5R5BPBWBPC6J42IA, seg_pos={0 0}, upload=519486]: Long tail cancellation due to optimal threshold Dec 30 08:03:12.318635588 (592h34m12.189718633s) [seg_pos={0 0}, upload=519486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:03:12.318744445 (592h34m12.18982749s) [upload=519486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:03:12.318752069 (592h34m12.189835114s) [upload=519486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:03:12.612436200 (592h34m12.483519245s) [upload=519486]: Done uploading segments... Dec 30 08:03:12.612461288 (592h34m12.483544323s) [upload=519486]: Upload finished. err: <nil> Duration: 5.713499482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 35.240185325s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:03:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 35.240215391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XXX....X....X.XX..X....XX.XX..................X...X..X..X] duration: 30.003007936s trace: https://jaeger.dev.storj.io/trace/174b4a72dc276c8e FAIL (2025-12-30 08:03:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: 5O7ZYTMX52MMTXLKNWYMNYDDIGXMAPIL77TJSZJAUBFQHZ4MDVFA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: RHPNY6DGOFKWIIUWFPAVYUPGD5SINESJ7AUI5YJJTPUTBKWTD4OQ) context deadline exceeded ======LOGS====== Dec 30 08:03:09.698527656 (592h34m9.569610691s) [upload=519486]: Starting upload Dec 30 08:03:09.698582189 (592h34m9.569665235s) [upload=519486]: Done starting upload Dec 30 08:03:09.698755720 (592h34m9.569838755s) [upload=519486]: Uploading segments... Dec 30 08:03:09.698788833 (592h34m9.569871868s) [upload=519486]: Got next segment. Inline: false Dec 30 08:03:09.698813390 (592h34m9.569896435s) [upload=519486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:03:09.698817608 (592h34m9.569900653s) [upload=519486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:03:09.951847797 (592h34m9.822930832s) [seg_pos={0 0}, upload=519486]: Begin upload segment... Dec 30 08:03:09.951891340 (592h34m9.822974375s) [seg_pos={0 0}, upload=519486]: Done begin upload segment. Dec 30 08:03:09.951917099 (592h34m9.823000134s) [noise=true , address=45.91.116.4:4940 , piece_id=FR5NT7NITJEDQUV5WWI7XU4IRYZ77S5OW4N4Y2HCZLH3UFSVJ4CQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.956278684 (592h34m9.827361719s) [noise=true , address=145.79.227.3:4151 , piece_id=WCNZYT5AGKC4VUESFZJMBOZXD5DXZOZBD55G6I3YRZCKU7DWZSBQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.957651439 (592h34m9.828734474s) [noise=true , address=45.91.116.14:4189 , piece_id=N6UOCLGM4ESTGLM5VO35RFZIIUX7MJVDOAPNZOJICZIER3CN4HKQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.958991560 (592h34m9.830074605s) [noise=true , address=145.79.227.22:4510 , piece_id=DVLQGVMACOGOPJBZ6ILHWH2QL3RA73T567EAXOF5UOVXAA5X2NJA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.960336431 (592h34m9.831419466s) [noise=true , address=145.79.227.23:4546 , piece_id=6LFZ7IQQREVOTAJ5ZV6X6E7SUEXGFFKR2Q6AS62GE7QANHIFBWDA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.961615577 (592h34m9.832698622s) [noise=true , address=145.79.227.16:4910 , piece_id=Q45RRY3XPAZV6F6BQB7FLXK4GFU7RX3EJPA3VA2S6PXK4L5TKN6A, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.963400195 (592h34m9.83448324s) [noise=true , address=145.79.227.32:4834 , piece_id=NUU2OTVN3KEOWLZ4VQ2R3DOHMTBKPDNLHRXL4TD2QWEZ4MRCJA4A, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.964939095 (592h34m9.836022141s) [noise=true , address=45.91.116.13:4156 , piece_id=A252VG3RXAOI3FNNROHPVU4GM7W7A224VG3Z5R5BPBWBPC6J42IA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.966446816 (592h34m9.837529861s) [noise=true , address=145.79.227.13:4802 , piece_id=C44TZGSPG47MHDKTOMTZUBHTTYH2XWQ3GDIHV2PBA2NLE3V4YM7A, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.968154818 (592h34m9.839237853s) [noise=true , address=145.79.227.11:4732 , piece_id=XHAGS7SZSUSXSMPG25J2ZSSF5ALU7ACMWOJDA37V2W2ES2FLU5RQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.969579672 (592h34m9.840662707s) [noise=true , address=77.90.31.17:4773 , piece_id=LSYUMJ4EHV5Y74UABQCIPKOIIGN25AH2463VBQVMMYX5W37N5JVQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.971111428 (592h34m9.842194473s) [noise=true , address=77.90.31.14:4661 , piece_id=5O7ZYTMX52MMTXLKNWYMNYDDIGXMAPIL77TJSZJAUBFQHZ4MDVFA, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.974254504 (592h34m9.845337549s) [noise=true , address=145.79.227.37:4566 , piece_id=RBN26VQGCFCWFMD3LJK6JXLYIVDUUCUROZA3ZFNFKBPNMDEQTC7Q, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.976059310 (592h34m9.847142355s) [noise=true , address=77.90.31.11:4555 , piece_id=RHPNY6DGOFKWIIUWFPAVYUPGD5SINESJ7AUI5YJJTPUTBKWTD4OQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.977694343 (592h34m9.848777378s) [noise=true , address=45.91.116.6:4975 , piece_id=2QG26HVQS3O4B4QZJ3ASJD27OG2V3EMEAHDSWJRJIGOSQV2VWF2Q, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:09.978997956 (592h34m9.850081001s) [noise=true , address=145.79.227.39:4014 , piece_id=2V2OJTOX7QNBZP6FJDVGN2UIYECGDNV4JHBNN4TTJCIPBZN5PTPQ, seg_pos={0 0}, upload=519486]: Uploading piece... Dec 30 08:03:11.704904506 (592h34m11.575987541s) [upload=519486]: Next returned nil segment Dec 30 08:03:11.704920005 (592h34m11.576003051s) [upload=519486]: Waiting for error group managing segments... Dec 30 08:03:11.989912006 (592h34m11.860995041s) [noise=true , address=145.79.227.13:4802 , piece_id=C44TZGSPG47MHDKTOMTZUBHTTYH2XWQ3GDIHV2PBA2NLE3V4YM7A, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:11.995741767 (592h34m11.866824802s) [noise=true , address=145.79.227.3:4151 , piece_id=WCNZYT5AGKC4VUESFZJMBOZXD5DXZOZBD55G6I3YRZCKU7DWZSBQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:11.996367138 (592h34m11.867450173s) [noise=true , address=145.79.227.39:4014 , piece_id=2V2OJTOX7QNBZP6FJDVGN2UIYECGDNV4JHBNN4TTJCIPBZN5PTPQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.001786587 (592h34m11.872869622s) [noise=true , address=145.79.227.22:4510 , piece_id=DVLQGVMACOGOPJBZ6ILHWH2QL3RA73T567EAXOF5UOVXAA5X2NJA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.017945095 (592h34m11.88902813s) [noise=true , address=145.79.227.16:4910 , piece_id=Q45RRY3XPAZV6F6BQB7FLXK4GFU7RX3EJPA3VA2S6PXK4L5TKN6A, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.022459421 (592h34m11.893542467s) [noise=true , address=145.79.227.32:4834 , piece_id=NUU2OTVN3KEOWLZ4VQ2R3DOHMTBKPDNLHRXL4TD2QWEZ4MRCJA4A, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.026947017 (592h34m11.898030052s) [noise=true , address=145.79.227.37:4566 , piece_id=RBN26VQGCFCWFMD3LJK6JXLYIVDUUCUROZA3ZFNFKBPNMDEQTC7Q, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.073022111 (592h34m11.944105156s) [noise=true , address=145.79.227.11:4732 , piece_id=XHAGS7SZSUSXSMPG25J2ZSSF5ALU7ACMWOJDA37V2W2ES2FLU5RQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.073930170 (592h34m11.945013215s) [noise=true , address=145.79.227.23:4546 , piece_id=6LFZ7IQQREVOTAJ5ZV6X6E7SUEXGFFKR2Q6AS62GE7QANHIFBWDA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.153017955 (592h34m12.02410103s) [noise=true , address=77.90.31.11:4555 , piece_id=RHPNY6DGOFKWIIUWFPAVYUPGD5SINESJ7AUI5YJJTPUTBKWTD4OQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.233887934 (592h34m12.104970979s) [noise=true , address=77.90.31.17:4773 , piece_id=LSYUMJ4EHV5Y74UABQCIPKOIIGN25AH2463VBQVMMYX5W37N5JVQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.285404190 (592h34m12.156487235s) [noise=true , address=77.90.31.14:4661 , piece_id=5O7ZYTMX52MMTXLKNWYMNYDDIGXMAPIL77TJSZJAUBFQHZ4MDVFA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.317923983 (592h34m12.189007018s) [noise=true , address=45.91.116.6:4975 , piece_id=2QG26HVQS3O4B4QZJ3ASJD27OG2V3EMEAHDSWJRJIGOSQV2VWF2Q, seg_pos={0 0}, upload=519486]: Done uploading piece. err: <nil> Dec 30 08:03:12.317950723 (592h34m12.189033758s) [seg_pos={0 0}, upload=519486]: Segment reached optimal threshold of 13 pieces. Dec 30 08:03:12.318216570 (592h34m12.189299615s) [noise=true , address=45.91.116.14:4189 , piece_id=N6UOCLGM4ESTGLM5VO35RFZIIUX7MJVDOAPNZOJICZIER3CN4HKQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:55238->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:55238->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55238->45.91.116.14:4189: use of closed network connection Dec 30 08:03:12.318227741 (592h34m12.189310776s) [noise=true , address=45.91.116.14:4189 , piece_id=N6UOCLGM4ESTGLM5VO35RFZIIUX7MJVDOAPNZOJICZIER3CN4HKQ, seg_pos={0 0}, upload=519486]: Long tail cancellation due to optimal threshold Dec 30 08:03:12.318469020 (592h34m12.189552056s) [noise=true , address=45.91.116.4:4940 , piece_id=FR5NT7NITJEDQUV5WWI7XU4IRYZ77S5OW4N4Y2HCZLH3UFSVJ4CQ, seg_pos={0 0}, upload=519486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 08:03:12.318490882 (592h34m12.189573928s) [noise=true , address=45.91.116.4:4940 , piece_id=FR5NT7NITJEDQUV5WWI7XU4IRYZ77S5OW4N4Y2HCZLH3UFSVJ4CQ, seg_pos={0 0}, upload=519486]: Long tail cancellation due to optimal threshold Dec 30 08:03:12.318563501 (592h34m12.189646536s) [noise=true , address=45.91.116.13:4156 , piece_id=A252VG3RXAOI3FNNROHPVU4GM7W7A224VG3Z5R5BPBWBPC6J42IA, seg_pos={0 0}, upload=519486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 08:03:12.318573269 (592h34m12.189656314s) [noise=true , address=45.91.116.13:4156 , piece_id=A252VG3RXAOI3FNNROHPVU4GM7W7A224VG3Z5R5BPBWBPC6J42IA, seg_pos={0 0}, upload=519486]: Long tail cancellation due to optimal threshold Dec 30 08:03:12.318635588 (592h34m12.189718633s) [seg_pos={0 0}, upload=519486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:03:12.318744445 (592h34m12.18982749s) [upload=519486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:03:12.318752069 (592h34m12.189835114s) [upload=519486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:03:12.612436200 (592h34m12.483519245s) [upload=519486]: Done uploading segments... Dec 30 08:03:12.612461288 (592h34m12.483544323s) [upload=519486]: Upload finished. err: <nil> Duration: 30.003033694s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X..] duration: 4.847852551s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:03:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.847882448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 20.569943426s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 20.569959216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XXX....X....X.XX..X....XX.XX..................X...X..X..] duration: 27.342475342s trace: https://jaeger.dev.storj.io/trace/2757fea082ee2294 FAIL (2025-12-30 08:03:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 08:00:58.136997126 (592h31m58.008080161s) [upload=519450]: Starting upload Dec 30 08:00:58.137073000 (592h31m58.008156035s) [upload=519450]: Done starting upload Dec 30 08:00:58.137271839 (592h31m58.008354874s) [upload=519450]: Uploading segments... Dec 30 08:00:58.137313068 (592h31m58.008396113s) [upload=519450]: Got next segment. Inline: false Dec 30 08:00:58.137344097 (592h31m58.008427152s) [upload=519450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:00:58.137349597 (592h31m58.008432652s) [upload=519450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:00:58.387840162 (592h31m58.258923217s) [seg_pos={0 0}, upload=519450]: Begin upload segment... Dec 30 08:00:58.387892691 (592h31m58.258975726s) [seg_pos={0 0}, upload=519450]: Done begin upload segment. Dec 30 08:00:58.387923670 (592h31m58.259006715s) [noise=true , address=145.79.227.11:4729 , piece_id=COJ3X2OM6WWXIKQRCMAF7TQZ4XGKLU6PXRIPVIJD4DKUCELWMAEA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.387929050 (592h31m58.259012095s) [noise=true , address=45.91.116.7:4009 , piece_id=ET7U7W4VBPGTIOTYTNCSYFUJSL5DCAMPPK6VL6QUOWD3QU75IRMQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.388386471 (592h31m58.259469506s) [noise=true , address=145.79.227.14:4837 , piece_id=FCL34DYY2TMBKFBIYL47LP5SBFCUS53774TLRRLABUQFSVDCGASA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.388775162 (592h31m58.259858207s) [noise=true , address=145.79.227.4:4187 , piece_id=RHMYOGH4NYKP7DRMVC7N6RNNSMQ443KORS6IKZZROAKEPYRNBZ2Q, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.389123395 (592h31m58.26020643s) [noise=true , address=77.90.31.18:4805 , piece_id=5DHOYOY5NKHQ2B56G5FD75QZR5P5KPGI2RIVQQ6USJSZF32N266Q, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.389489662 (592h31m58.260572697s) [noise=true , address=45.91.116.9:4264 , piece_id=6HYD5H6Z5FH3YTN4H5IWNVX3LDLZZABOBZCPI7HMEDGUEXI3CDOA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.389970368 (592h31m58.261053403s) [noise=true , address=145.79.227.36:4517 , piece_id=KPCDGYZB4P3BCKUYOEV5XY5P7XWXXFFI3PZ56H26LHS7UAVNAS5Q, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.390276350 (592h31m58.261359396s) [noise=true , address=145.79.227.23:4546 , piece_id=2HWSVWSDQOHD55AHBR3QCN3HQIUWOCE5TNYO7TTQ2I4OHYJF26DA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.390624453 (592h31m58.261707498s) [noise=true , address=145.79.227.42:4123 , piece_id=7NJTOLZ5VD35PDK5A2C7ZTVN7WQAJVEJF2RVUHWYEEFZ2KJGBD6A, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.391126098 (592h31m58.262209143s) [noise=true , address=145.79.227.15:4874 , piece_id=BZ7MTV6KQ5FSEJZCAL6GIDLNUXZLVMFRXCMJVPWB4WDPTWELFHZQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.391501263 (592h31m58.262584308s) [noise=true , address=145.79.227.19:4077 , piece_id=ZXOJURPNOUUKXUF6GAZ3UXWYNWKCU5DYMRTB535HWD36JULOWYCA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.393231047 (592h31m58.264314092s) [noise=true , address=145.79.227.16:4912 , piece_id=MVGM2I6R7JMGKANOUVKR6PBQYNPHWFW3A7RQ6YOI5QL3MMG42VYA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.394724952 (592h31m58.265807987s) [noise=true , address=145.79.227.37:4566 , piece_id=VDQLS5KMV6SXTEYEOSQVUSVL2CC6PG334R5U4LQS7CJ7GP4XCNNQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.396062208 (592h31m58.267145243s) [noise=true , address=145.79.227.25:4617 , piece_id=7EUY672OIBASWILOZZFP56A62VCXQWKP7YQOT2JWQFA23HXIJQLQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.397295596 (592h31m58.268378641s) [noise=true , address=77.90.31.21:4877 , piece_id=TYOG2GTG6T52CKAZIGAK3FP32V64T5CWQPRVOUCM6CI6FUHKAESA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.398975846 (592h31m58.270058891s) [noise=true , address=145.79.227.33:4869 , piece_id=UXK6BXCH2FH7MZGJXN3LQXJ24CVJWXREPUYT2FODQ6PRYLNEA7MQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:01:00.113331381 (592h31m59.984414416s) [upload=519450]: Next returned nil segment Dec 30 08:01:00.113369824 (592h31m59.984452869s) [upload=519450]: Waiting for error group managing segments... Dec 30 08:01:00.385739830 (592h32m0.256822875s) [noise=true , address=145.79.227.14:4837 , piece_id=FCL34DYY2TMBKFBIYL47LP5SBFCUS53774TLRRLABUQFSVDCGASA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.395722549 (592h32m0.266805594s) [noise=true , address=145.79.227.4:4187 , piece_id=RHMYOGH4NYKP7DRMVC7N6RNNSMQ443KORS6IKZZROAKEPYRNBZ2Q, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.404212075 (592h32m0.27529512s) [noise=true , address=145.79.227.15:4874 , piece_id=BZ7MTV6KQ5FSEJZCAL6GIDLNUXZLVMFRXCMJVPWB4WDPTWELFHZQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.407450501 (592h32m0.278533537s) [noise=true , address=145.79.227.42:4123 , piece_id=7NJTOLZ5VD35PDK5A2C7ZTVN7WQAJVEJF2RVUHWYEEFZ2KJGBD6A, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.409535792 (592h32m0.280618827s) [noise=true , address=145.79.227.16:4912 , piece_id=MVGM2I6R7JMGKANOUVKR6PBQYNPHWFW3A7RQ6YOI5QL3MMG42VYA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.411251820 (592h32m0.282334865s) [noise=true , address=145.79.227.11:4729 , piece_id=COJ3X2OM6WWXIKQRCMAF7TQZ4XGKLU6PXRIPVIJD4DKUCELWMAEA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.414703572 (592h32m0.285786617s) [noise=true , address=145.79.227.19:4077 , piece_id=ZXOJURPNOUUKXUF6GAZ3UXWYNWKCU5DYMRTB535HWD36JULOWYCA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.419973226 (592h32m0.291056271s) [noise=true , address=145.79.227.36:4517 , piece_id=KPCDGYZB4P3BCKUYOEV5XY5P7XWXXFFI3PZ56H26LHS7UAVNAS5Q, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.426797972 (592h32m0.297881017s) [noise=true , address=145.79.227.23:4546 , piece_id=2HWSVWSDQOHD55AHBR3QCN3HQIUWOCE5TNYO7TTQ2I4OHYJF26DA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.475321287 (592h32m0.346404322s) [noise=true , address=145.79.227.37:4566 , piece_id=VDQLS5KMV6SXTEYEOSQVUSVL2CC6PG334R5U4LQS7CJ7GP4XCNNQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.557886204 (592h32m0.42896925s) [noise=true , address=145.79.227.25:4617 , piece_id=7EUY672OIBASWILOZZFP56A62VCXQWKP7YQOT2JWQFA23HXIJQLQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.692390482 (592h32m0.563473517s) [noise=true , address=77.90.31.18:4805 , piece_id=5DHOYOY5NKHQ2B56G5FD75QZR5P5KPGI2RIVQQ6USJSZF32N266Q, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.714005220 (592h32m0.585088285s) [noise=true , address=45.91.116.7:4009 , piece_id=ET7U7W4VBPGTIOTYTNCSYFUJSL5DCAMPPK6VL6QUOWD3QU75IRMQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.714043573 (592h32m0.585126618s) [seg_pos={0 0}, upload=519450]: Segment reached optimal threshold of 13 pieces. Dec 30 08:01:00.714602988 (592h32m0.585686033s) [noise=true , address=145.79.227.33:4869 , piece_id=UXK6BXCH2FH7MZGJXN3LQXJ24CVJWXREPUYT2FODQ6PRYLNEA7MQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 08:01:00.714612937 (592h32m0.585695982s) [noise=true , address=145.79.227.33:4869 , piece_id=UXK6BXCH2FH7MZGJXN3LQXJ24CVJWXREPUYT2FODQ6PRYLNEA7MQ, seg_pos={0 0}, upload=519450]: Long tail cancellation due to optimal threshold Dec 30 08:01:00.714962322 (592h32m0.586045427s) [noise=true , address=77.90.31.21:4877 , piece_id=TYOG2GTG6T52CKAZIGAK3FP32V64T5CWQPRVOUCM6CI6FUHKAESA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:46748->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 08:01:00.714973745 (592h32m0.58605679s) [noise=true , address=77.90.31.21:4877 , piece_id=TYOG2GTG6T52CKAZIGAK3FP32V64T5CWQPRVOUCM6CI6FUHKAESA, seg_pos={0 0}, upload=519450]: Long tail cancellation due to optimal threshold Dec 30 08:01:00.715316467 (592h32m0.586399523s) [noise=true , address=45.91.116.9:4264 , piece_id=6HYD5H6Z5FH3YTN4H5IWNVX3LDLZZABOBZCPI7HMEDGUEXI3CDOA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 08:01:00.715326596 (592h32m0.586409641s) [noise=true , address=45.91.116.9:4264 , piece_id=6HYD5H6Z5FH3YTN4H5IWNVX3LDLZZABOBZCPI7HMEDGUEXI3CDOA, seg_pos={0 0}, upload=519450]: Long tail cancellation due to optimal threshold Dec 30 08:01:00.715346404 (592h32m0.586429469s) [seg_pos={0 0}, upload=519450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:01:00.715438099 (592h32m0.586521144s) [upload=519450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:01:00.715444631 (592h32m0.586527676s) [upload=519450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:01:01.015622708 (592h32m0.886705753s) [upload=519450]: Done uploading segments... Dec 30 08:01:01.015643918 (592h32m0.886726963s) [upload=519450]: Upload finished. err: <nil> Duration: 27.342498968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 28.348165035s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:02:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 28.348182961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 30.228762636s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:02:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 30.228788224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X.] duration: 3.333915095s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:02:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.33394964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X....XXX....X....X.XX..X....XX.XX..................X...X..X.] duration: 6.773707738s trace: https://jaeger.dev.storj.io/trace/2757fea082ee2294 FAIL (2025-12-30 08:01:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 08:00:58.136997126 (592h31m58.008080161s) [upload=519450]: Starting upload Dec 30 08:00:58.137073000 (592h31m58.008156035s) [upload=519450]: Done starting upload Dec 30 08:00:58.137271839 (592h31m58.008354874s) [upload=519450]: Uploading segments... Dec 30 08:00:58.137313068 (592h31m58.008396113s) [upload=519450]: Got next segment. Inline: false Dec 30 08:00:58.137344097 (592h31m58.008427152s) [upload=519450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:00:58.137349597 (592h31m58.008432652s) [upload=519450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:00:58.387840162 (592h31m58.258923217s) [seg_pos={0 0}, upload=519450]: Begin upload segment... Dec 30 08:00:58.387892691 (592h31m58.258975726s) [seg_pos={0 0}, upload=519450]: Done begin upload segment. Dec 30 08:00:58.387923670 (592h31m58.259006715s) [noise=true , address=145.79.227.11:4729 , piece_id=COJ3X2OM6WWXIKQRCMAF7TQZ4XGKLU6PXRIPVIJD4DKUCELWMAEA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.387929050 (592h31m58.259012095s) [noise=true , address=45.91.116.7:4009 , piece_id=ET7U7W4VBPGTIOTYTNCSYFUJSL5DCAMPPK6VL6QUOWD3QU75IRMQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.388386471 (592h31m58.259469506s) [noise=true , address=145.79.227.14:4837 , piece_id=FCL34DYY2TMBKFBIYL47LP5SBFCUS53774TLRRLABUQFSVDCGASA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.388775162 (592h31m58.259858207s) [noise=true , address=145.79.227.4:4187 , piece_id=RHMYOGH4NYKP7DRMVC7N6RNNSMQ443KORS6IKZZROAKEPYRNBZ2Q, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.389123395 (592h31m58.26020643s) [noise=true , address=77.90.31.18:4805 , piece_id=5DHOYOY5NKHQ2B56G5FD75QZR5P5KPGI2RIVQQ6USJSZF32N266Q, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.389489662 (592h31m58.260572697s) [noise=true , address=45.91.116.9:4264 , piece_id=6HYD5H6Z5FH3YTN4H5IWNVX3LDLZZABOBZCPI7HMEDGUEXI3CDOA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.389970368 (592h31m58.261053403s) [noise=true , address=145.79.227.36:4517 , piece_id=KPCDGYZB4P3BCKUYOEV5XY5P7XWXXFFI3PZ56H26LHS7UAVNAS5Q, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.390276350 (592h31m58.261359396s) [noise=true , address=145.79.227.23:4546 , piece_id=2HWSVWSDQOHD55AHBR3QCN3HQIUWOCE5TNYO7TTQ2I4OHYJF26DA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.390624453 (592h31m58.261707498s) [noise=true , address=145.79.227.42:4123 , piece_id=7NJTOLZ5VD35PDK5A2C7ZTVN7WQAJVEJF2RVUHWYEEFZ2KJGBD6A, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.391126098 (592h31m58.262209143s) [noise=true , address=145.79.227.15:4874 , piece_id=BZ7MTV6KQ5FSEJZCAL6GIDLNUXZLVMFRXCMJVPWB4WDPTWELFHZQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.391501263 (592h31m58.262584308s) [noise=true , address=145.79.227.19:4077 , piece_id=ZXOJURPNOUUKXUF6GAZ3UXWYNWKCU5DYMRTB535HWD36JULOWYCA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.393231047 (592h31m58.264314092s) [noise=true , address=145.79.227.16:4912 , piece_id=MVGM2I6R7JMGKANOUVKR6PBQYNPHWFW3A7RQ6YOI5QL3MMG42VYA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.394724952 (592h31m58.265807987s) [noise=true , address=145.79.227.37:4566 , piece_id=VDQLS5KMV6SXTEYEOSQVUSVL2CC6PG334R5U4LQS7CJ7GP4XCNNQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.396062208 (592h31m58.267145243s) [noise=true , address=145.79.227.25:4617 , piece_id=7EUY672OIBASWILOZZFP56A62VCXQWKP7YQOT2JWQFA23HXIJQLQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.397295596 (592h31m58.268378641s) [noise=true , address=77.90.31.21:4877 , piece_id=TYOG2GTG6T52CKAZIGAK3FP32V64T5CWQPRVOUCM6CI6FUHKAESA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.398975846 (592h31m58.270058891s) [noise=true , address=145.79.227.33:4869 , piece_id=UXK6BXCH2FH7MZGJXN3LQXJ24CVJWXREPUYT2FODQ6PRYLNEA7MQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:01:00.113331381 (592h31m59.984414416s) [upload=519450]: Next returned nil segment Dec 30 08:01:00.113369824 (592h31m59.984452869s) [upload=519450]: Waiting for error group managing segments... Dec 30 08:01:00.385739830 (592h32m0.256822875s) [noise=true , address=145.79.227.14:4837 , piece_id=FCL34DYY2TMBKFBIYL47LP5SBFCUS53774TLRRLABUQFSVDCGASA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.395722549 (592h32m0.266805594s) [noise=true , address=145.79.227.4:4187 , piece_id=RHMYOGH4NYKP7DRMVC7N6RNNSMQ443KORS6IKZZROAKEPYRNBZ2Q, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.404212075 (592h32m0.27529512s) [noise=true , address=145.79.227.15:4874 , piece_id=BZ7MTV6KQ5FSEJZCAL6GIDLNUXZLVMFRXCMJVPWB4WDPTWELFHZQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.407450501 (592h32m0.278533537s) [noise=true , address=145.79.227.42:4123 , piece_id=7NJTOLZ5VD35PDK5A2C7ZTVN7WQAJVEJF2RVUHWYEEFZ2KJGBD6A, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.409535792 (592h32m0.280618827s) [noise=true , address=145.79.227.16:4912 , piece_id=MVGM2I6R7JMGKANOUVKR6PBQYNPHWFW3A7RQ6YOI5QL3MMG42VYA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.411251820 (592h32m0.282334865s) [noise=true , address=145.79.227.11:4729 , piece_id=COJ3X2OM6WWXIKQRCMAF7TQZ4XGKLU6PXRIPVIJD4DKUCELWMAEA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.414703572 (592h32m0.285786617s) [noise=true , address=145.79.227.19:4077 , piece_id=ZXOJURPNOUUKXUF6GAZ3UXWYNWKCU5DYMRTB535HWD36JULOWYCA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.419973226 (592h32m0.291056271s) [noise=true , address=145.79.227.36:4517 , piece_id=KPCDGYZB4P3BCKUYOEV5XY5P7XWXXFFI3PZ56H26LHS7UAVNAS5Q, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.426797972 (592h32m0.297881017s) [noise=true , address=145.79.227.23:4546 , piece_id=2HWSVWSDQOHD55AHBR3QCN3HQIUWOCE5TNYO7TTQ2I4OHYJF26DA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.475321287 (592h32m0.346404322s) [noise=true , address=145.79.227.37:4566 , piece_id=VDQLS5KMV6SXTEYEOSQVUSVL2CC6PG334R5U4LQS7CJ7GP4XCNNQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.557886204 (592h32m0.42896925s) [noise=true , address=145.79.227.25:4617 , piece_id=7EUY672OIBASWILOZZFP56A62VCXQWKP7YQOT2JWQFA23HXIJQLQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.692390482 (592h32m0.563473517s) [noise=true , address=77.90.31.18:4805 , piece_id=5DHOYOY5NKHQ2B56G5FD75QZR5P5KPGI2RIVQQ6USJSZF32N266Q, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.714005220 (592h32m0.585088285s) [noise=true , address=45.91.116.7:4009 , piece_id=ET7U7W4VBPGTIOTYTNCSYFUJSL5DCAMPPK6VL6QUOWD3QU75IRMQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.714043573 (592h32m0.585126618s) [seg_pos={0 0}, upload=519450]: Segment reached optimal threshold of 13 pieces. Dec 30 08:01:00.714602988 (592h32m0.585686033s) [noise=true , address=145.79.227.33:4869 , piece_id=UXK6BXCH2FH7MZGJXN3LQXJ24CVJWXREPUYT2FODQ6PRYLNEA7MQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 08:01:00.714612937 (592h32m0.585695982s) [noise=true , address=145.79.227.33:4869 , piece_id=UXK6BXCH2FH7MZGJXN3LQXJ24CVJWXREPUYT2FODQ6PRYLNEA7MQ, seg_pos={0 0}, upload=519450]: Long tail cancellation due to optimal threshold Dec 30 08:01:00.714962322 (592h32m0.586045427s) [noise=true , address=77.90.31.21:4877 , piece_id=TYOG2GTG6T52CKAZIGAK3FP32V64T5CWQPRVOUCM6CI6FUHKAESA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:46748->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 08:01:00.714973745 (592h32m0.58605679s) [noise=true , address=77.90.31.21:4877 , piece_id=TYOG2GTG6T52CKAZIGAK3FP32V64T5CWQPRVOUCM6CI6FUHKAESA, seg_pos={0 0}, upload=519450]: Long tail cancellation due to optimal threshold Dec 30 08:01:00.715316467 (592h32m0.586399523s) [noise=true , address=45.91.116.9:4264 , piece_id=6HYD5H6Z5FH3YTN4H5IWNVX3LDLZZABOBZCPI7HMEDGUEXI3CDOA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 08:01:00.715326596 (592h32m0.586409641s) [noise=true , address=45.91.116.9:4264 , piece_id=6HYD5H6Z5FH3YTN4H5IWNVX3LDLZZABOBZCPI7HMEDGUEXI3CDOA, seg_pos={0 0}, upload=519450]: Long tail cancellation due to optimal threshold Dec 30 08:01:00.715346404 (592h32m0.586429469s) [seg_pos={0 0}, upload=519450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:01:00.715438099 (592h32m0.586521144s) [upload=519450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:01:00.715444631 (592h32m0.586527676s) [upload=519450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:01:01.015622708 (592h32m0.886705753s) [upload=519450]: Done uploading segments... Dec 30 08:01:01.015643918 (592h32m0.886726963s) [upload=519450]: Upload finished. err: <nil> Duration: 6.773737325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 13.866772308s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:01:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 13.866800221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [..X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..X] duration: 5.001200722s trace: https://jaeger.dev.storj.io/trace/6cbf3c85aed6fd64 FAIL (2025-12-30 08:01:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001217504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X....XXX....X....X.XX..X....XX.XX..................X...X..X] duration: 30.002975022s trace: https://jaeger.dev.storj.io/trace/2757fea082ee2294 FAIL (2025-12-30 08:01:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 08:00:58.136997126 (592h31m58.008080161s) [upload=519450]: Starting upload Dec 30 08:00:58.137073000 (592h31m58.008156035s) [upload=519450]: Done starting upload Dec 30 08:00:58.137271839 (592h31m58.008354874s) [upload=519450]: Uploading segments... Dec 30 08:00:58.137313068 (592h31m58.008396113s) [upload=519450]: Got next segment. Inline: false Dec 30 08:00:58.137344097 (592h31m58.008427152s) [upload=519450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 08:00:58.137349597 (592h31m58.008432652s) [upload=519450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 08:00:58.387840162 (592h31m58.258923217s) [seg_pos={0 0}, upload=519450]: Begin upload segment... Dec 30 08:00:58.387892691 (592h31m58.258975726s) [seg_pos={0 0}, upload=519450]: Done begin upload segment. Dec 30 08:00:58.387923670 (592h31m58.259006715s) [noise=true , address=145.79.227.11:4729 , piece_id=COJ3X2OM6WWXIKQRCMAF7TQZ4XGKLU6PXRIPVIJD4DKUCELWMAEA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.387929050 (592h31m58.259012095s) [noise=true , address=45.91.116.7:4009 , piece_id=ET7U7W4VBPGTIOTYTNCSYFUJSL5DCAMPPK6VL6QUOWD3QU75IRMQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.388386471 (592h31m58.259469506s) [noise=true , address=145.79.227.14:4837 , piece_id=FCL34DYY2TMBKFBIYL47LP5SBFCUS53774TLRRLABUQFSVDCGASA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.388775162 (592h31m58.259858207s) [noise=true , address=145.79.227.4:4187 , piece_id=RHMYOGH4NYKP7DRMVC7N6RNNSMQ443KORS6IKZZROAKEPYRNBZ2Q, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.389123395 (592h31m58.26020643s) [noise=true , address=77.90.31.18:4805 , piece_id=5DHOYOY5NKHQ2B56G5FD75QZR5P5KPGI2RIVQQ6USJSZF32N266Q, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.389489662 (592h31m58.260572697s) [noise=true , address=45.91.116.9:4264 , piece_id=6HYD5H6Z5FH3YTN4H5IWNVX3LDLZZABOBZCPI7HMEDGUEXI3CDOA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.389970368 (592h31m58.261053403s) [noise=true , address=145.79.227.36:4517 , piece_id=KPCDGYZB4P3BCKUYOEV5XY5P7XWXXFFI3PZ56H26LHS7UAVNAS5Q, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.390276350 (592h31m58.261359396s) [noise=true , address=145.79.227.23:4546 , piece_id=2HWSVWSDQOHD55AHBR3QCN3HQIUWOCE5TNYO7TTQ2I4OHYJF26DA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.390624453 (592h31m58.261707498s) [noise=true , address=145.79.227.42:4123 , piece_id=7NJTOLZ5VD35PDK5A2C7ZTVN7WQAJVEJF2RVUHWYEEFZ2KJGBD6A, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.391126098 (592h31m58.262209143s) [noise=true , address=145.79.227.15:4874 , piece_id=BZ7MTV6KQ5FSEJZCAL6GIDLNUXZLVMFRXCMJVPWB4WDPTWELFHZQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.391501263 (592h31m58.262584308s) [noise=true , address=145.79.227.19:4077 , piece_id=ZXOJURPNOUUKXUF6GAZ3UXWYNWKCU5DYMRTB535HWD36JULOWYCA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.393231047 (592h31m58.264314092s) [noise=true , address=145.79.227.16:4912 , piece_id=MVGM2I6R7JMGKANOUVKR6PBQYNPHWFW3A7RQ6YOI5QL3MMG42VYA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.394724952 (592h31m58.265807987s) [noise=true , address=145.79.227.37:4566 , piece_id=VDQLS5KMV6SXTEYEOSQVUSVL2CC6PG334R5U4LQS7CJ7GP4XCNNQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.396062208 (592h31m58.267145243s) [noise=true , address=145.79.227.25:4617 , piece_id=7EUY672OIBASWILOZZFP56A62VCXQWKP7YQOT2JWQFA23HXIJQLQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.397295596 (592h31m58.268378641s) [noise=true , address=77.90.31.21:4877 , piece_id=TYOG2GTG6T52CKAZIGAK3FP32V64T5CWQPRVOUCM6CI6FUHKAESA, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:00:58.398975846 (592h31m58.270058891s) [noise=true , address=145.79.227.33:4869 , piece_id=UXK6BXCH2FH7MZGJXN3LQXJ24CVJWXREPUYT2FODQ6PRYLNEA7MQ, seg_pos={0 0}, upload=519450]: Uploading piece... Dec 30 08:01:00.113331381 (592h31m59.984414416s) [upload=519450]: Next returned nil segment Dec 30 08:01:00.113369824 (592h31m59.984452869s) [upload=519450]: Waiting for error group managing segments... Dec 30 08:01:00.385739830 (592h32m0.256822875s) [noise=true , address=145.79.227.14:4837 , piece_id=FCL34DYY2TMBKFBIYL47LP5SBFCUS53774TLRRLABUQFSVDCGASA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.395722549 (592h32m0.266805594s) [noise=true , address=145.79.227.4:4187 , piece_id=RHMYOGH4NYKP7DRMVC7N6RNNSMQ443KORS6IKZZROAKEPYRNBZ2Q, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.404212075 (592h32m0.27529512s) [noise=true , address=145.79.227.15:4874 , piece_id=BZ7MTV6KQ5FSEJZCAL6GIDLNUXZLVMFRXCMJVPWB4WDPTWELFHZQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.407450501 (592h32m0.278533537s) [noise=true , address=145.79.227.42:4123 , piece_id=7NJTOLZ5VD35PDK5A2C7ZTVN7WQAJVEJF2RVUHWYEEFZ2KJGBD6A, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.409535792 (592h32m0.280618827s) [noise=true , address=145.79.227.16:4912 , piece_id=MVGM2I6R7JMGKANOUVKR6PBQYNPHWFW3A7RQ6YOI5QL3MMG42VYA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.411251820 (592h32m0.282334865s) [noise=true , address=145.79.227.11:4729 , piece_id=COJ3X2OM6WWXIKQRCMAF7TQZ4XGKLU6PXRIPVIJD4DKUCELWMAEA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.414703572 (592h32m0.285786617s) [noise=true , address=145.79.227.19:4077 , piece_id=ZXOJURPNOUUKXUF6GAZ3UXWYNWKCU5DYMRTB535HWD36JULOWYCA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.419973226 (592h32m0.291056271s) [noise=true , address=145.79.227.36:4517 , piece_id=KPCDGYZB4P3BCKUYOEV5XY5P7XWXXFFI3PZ56H26LHS7UAVNAS5Q, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.426797972 (592h32m0.297881017s) [noise=true , address=145.79.227.23:4546 , piece_id=2HWSVWSDQOHD55AHBR3QCN3HQIUWOCE5TNYO7TTQ2I4OHYJF26DA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.475321287 (592h32m0.346404322s) [noise=true , address=145.79.227.37:4566 , piece_id=VDQLS5KMV6SXTEYEOSQVUSVL2CC6PG334R5U4LQS7CJ7GP4XCNNQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.557886204 (592h32m0.42896925s) [noise=true , address=145.79.227.25:4617 , piece_id=7EUY672OIBASWILOZZFP56A62VCXQWKP7YQOT2JWQFA23HXIJQLQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.692390482 (592h32m0.563473517s) [noise=true , address=77.90.31.18:4805 , piece_id=5DHOYOY5NKHQ2B56G5FD75QZR5P5KPGI2RIVQQ6USJSZF32N266Q, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.714005220 (592h32m0.585088285s) [noise=true , address=45.91.116.7:4009 , piece_id=ET7U7W4VBPGTIOTYTNCSYFUJSL5DCAMPPK6VL6QUOWD3QU75IRMQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: <nil> Dec 30 08:01:00.714043573 (592h32m0.585126618s) [seg_pos={0 0}, upload=519450]: Segment reached optimal threshold of 13 pieces. Dec 30 08:01:00.714602988 (592h32m0.585686033s) [noise=true , address=145.79.227.33:4869 , piece_id=UXK6BXCH2FH7MZGJXN3LQXJ24CVJWXREPUYT2FODQ6PRYLNEA7MQ, seg_pos={0 0}, upload=519450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 08:01:00.714612937 (592h32m0.585695982s) [noise=true , address=145.79.227.33:4869 , piece_id=UXK6BXCH2FH7MZGJXN3LQXJ24CVJWXREPUYT2FODQ6PRYLNEA7MQ, seg_pos={0 0}, upload=519450]: Long tail cancellation due to optimal threshold Dec 30 08:01:00.714962322 (592h32m0.586045427s) [noise=true , address=77.90.31.21:4877 , piece_id=TYOG2GTG6T52CKAZIGAK3FP32V64T5CWQPRVOUCM6CI6FUHKAESA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: read tcp 188.166.225.170:46748->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 08:01:00.714973745 (592h32m0.58605679s) [noise=true , address=77.90.31.21:4877 , piece_id=TYOG2GTG6T52CKAZIGAK3FP32V64T5CWQPRVOUCM6CI6FUHKAESA, seg_pos={0 0}, upload=519450]: Long tail cancellation due to optimal threshold Dec 30 08:01:00.715316467 (592h32m0.586399523s) [noise=true , address=45.91.116.9:4264 , piece_id=6HYD5H6Z5FH3YTN4H5IWNVX3LDLZZABOBZCPI7HMEDGUEXI3CDOA, seg_pos={0 0}, upload=519450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 08:01:00.715326596 (592h32m0.586409641s) [noise=true , address=45.91.116.9:4264 , piece_id=6HYD5H6Z5FH3YTN4H5IWNVX3LDLZZABOBZCPI7HMEDGUEXI3CDOA, seg_pos={0 0}, upload=519450]: Long tail cancellation due to optimal threshold Dec 30 08:01:00.715346404 (592h32m0.586429469s) [seg_pos={0 0}, upload=519450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 08:01:00.715438099 (592h32m0.586521144s) [upload=519450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 08:01:00.715444631 (592h32m0.586527676s) [upload=519450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 08:01:01.015622708 (592h32m0.886705753s) [upload=519450]: Done uploading segments... Dec 30 08:01:01.015643918 (592h32m0.886726963s) [upload=519450]: Upload finished. err: <nil> Duration: 30.002996753s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [...X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X..] duration: 3.503220989s trace: https://jaeger.dev.storj.io/trace/781ac7d52d9c13dc FAIL (2025-12-30 08:01:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503273871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 44.491447322s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:01:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 44.491466868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....XXX....X....X.XX..X....XX.XX..................X...X..] duration: 6.460105304s trace: https://jaeger.dev.storj.io/trace/2d666678f6513905 FAIL (2025-12-30 08:00:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q) context deadline exceeded ======LOGS====== Dec 30 07:59:10.202653803 (592h30m10.073736848s) [upload=519423]: Starting upload Dec 30 07:59:10.202752560 (592h30m10.073835605s) [upload=519423]: Done starting upload Dec 30 07:59:10.202990363 (592h30m10.074073418s) [upload=519423]: Uploading segments... Dec 30 07:59:10.203134788 (592h30m10.074217833s) [upload=519423]: Got next segment. Inline: false Dec 30 07:59:10.203202307 (592h30m10.074285342s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:59:10.203208508 (592h30m10.074291553s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:59:10.447546036 (592h30m10.318629081s) [seg_pos={0 0}, upload=519423]: Begin upload segment... Dec 30 07:59:10.447642639 (592h30m10.318725684s) [seg_pos={0 0}, upload=519423]: Done begin upload segment. Dec 30 07:59:10.447706581 (592h30m10.318789626s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447713214 (592h30m10.318796249s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447924818 (592h30m10.319007863s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448485695 (592h30m10.31956873s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448958796 (592h30m10.320041841s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.449438610 (592h30m10.320521655s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450059773 (592h30m10.321142818s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450651269 (592h30m10.321734314s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.452118853 (592h30m10.323201898s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453400123 (592h30m10.324483168s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453661260 (592h30m10.324744305s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.455123524 (592h30m10.326206559s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.456465690 (592h30m10.327548726s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.458194782 (592h30m10.329277817s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.460128836 (592h30m10.331211881s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.462118644 (592h30m10.333201679s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:12.144433349 (592h30m12.015516395s) [upload=519423]: Next returned nil segment Dec 30 07:59:12.144446866 (592h30m12.015529911s) [upload=519423]: Waiting for error group managing segments... Dec 30 07:59:12.427594743 (592h30m12.298677788s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.433067574 (592h30m12.304150619s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.450044672 (592h30m12.321127717s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.456286157 (592h30m12.327369192s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.460033563 (592h30m12.331116598s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.463362521 (592h30m12.334445556s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.467905432 (592h30m12.338988467s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.468015401 (592h30m12.339098446s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.534010025 (592h30m12.405093061s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.553978358 (592h30m12.425061393s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.653048103 (592h30m12.524131158s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.876260555 (592h30m12.7473436s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265833353 (592h30m13.136916388s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265855445 (592h30m13.13693848s) [seg_pos={0 0}, upload=519423]: Segment reached optimal threshold of 13 pieces. Dec 30 07:59:13.266178440 (592h30m13.137261475s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:46954->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:59:13.266190233 (592h30m13.137273308s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266311203 (592h30m13.137394238s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266327554 (592h30m13.137410589s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266376197 (592h30m13.137459232s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266382398 (592h30m13.137465443s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266397808 (592h30m13.137480843s) [seg_pos={0 0}, upload=519423]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:59:13.266510162 (592h30m13.137593207s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:59:13.266518638 (592h30m13.137601673s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:59:13.555405394 (592h30m13.426488439s) [upload=519423]: Done uploading segments... Dec 30 07:59:13.555425743 (592h30m13.426508778s) [upload=519423]: Upload finished. err: <nil> Duration: 6.460335992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 42.607760659s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 08:00:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 42.607780587s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X.] duration: 4.367117796s trace: https://jaeger.dev.storj.io/trace/781ac7d52d9c13dc FAIL (2025-12-30 08:00:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.367148523s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..X] duration: 5.00223422s trace: https://jaeger.dev.storj.io/trace/781ac7d52d9c13dc FAIL (2025-12-30 08:00:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002259108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X....XXX....X....X.XX..X....XX.XX..................X...X.] duration: 5.912237752s trace: https://jaeger.dev.storj.io/trace/2d666678f6513905 FAIL (2025-12-30 07:59:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q) context deadline exceeded ======LOGS====== Dec 30 07:59:10.202653803 (592h30m10.073736848s) [upload=519423]: Starting upload Dec 30 07:59:10.202752560 (592h30m10.073835605s) [upload=519423]: Done starting upload Dec 30 07:59:10.202990363 (592h30m10.074073418s) [upload=519423]: Uploading segments... Dec 30 07:59:10.203134788 (592h30m10.074217833s) [upload=519423]: Got next segment. Inline: false Dec 30 07:59:10.203202307 (592h30m10.074285342s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:59:10.203208508 (592h30m10.074291553s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:59:10.447546036 (592h30m10.318629081s) [seg_pos={0 0}, upload=519423]: Begin upload segment... Dec 30 07:59:10.447642639 (592h30m10.318725684s) [seg_pos={0 0}, upload=519423]: Done begin upload segment. Dec 30 07:59:10.447706581 (592h30m10.318789626s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447713214 (592h30m10.318796249s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447924818 (592h30m10.319007863s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448485695 (592h30m10.31956873s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448958796 (592h30m10.320041841s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.449438610 (592h30m10.320521655s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450059773 (592h30m10.321142818s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450651269 (592h30m10.321734314s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.452118853 (592h30m10.323201898s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453400123 (592h30m10.324483168s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453661260 (592h30m10.324744305s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.455123524 (592h30m10.326206559s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.456465690 (592h30m10.327548726s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.458194782 (592h30m10.329277817s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.460128836 (592h30m10.331211881s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.462118644 (592h30m10.333201679s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:12.144433349 (592h30m12.015516395s) [upload=519423]: Next returned nil segment Dec 30 07:59:12.144446866 (592h30m12.015529911s) [upload=519423]: Waiting for error group managing segments... Dec 30 07:59:12.427594743 (592h30m12.298677788s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.433067574 (592h30m12.304150619s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.450044672 (592h30m12.321127717s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.456286157 (592h30m12.327369192s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.460033563 (592h30m12.331116598s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.463362521 (592h30m12.334445556s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.467905432 (592h30m12.338988467s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.468015401 (592h30m12.339098446s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.534010025 (592h30m12.405093061s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.553978358 (592h30m12.425061393s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.653048103 (592h30m12.524131158s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.876260555 (592h30m12.7473436s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265833353 (592h30m13.136916388s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265855445 (592h30m13.13693848s) [seg_pos={0 0}, upload=519423]: Segment reached optimal threshold of 13 pieces. Dec 30 07:59:13.266178440 (592h30m13.137261475s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:46954->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:59:13.266190233 (592h30m13.137273308s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266311203 (592h30m13.137394238s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266327554 (592h30m13.137410589s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266376197 (592h30m13.137459232s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266382398 (592h30m13.137465443s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266397808 (592h30m13.137480843s) [seg_pos={0 0}, upload=519423]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:59:13.266510162 (592h30m13.137593207s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:59:13.266518638 (592h30m13.137601673s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:59:13.555405394 (592h30m13.426488439s) [upload=519423]: Done uploading segments... Dec 30 07:59:13.555425743 (592h30m13.426508778s) [upload=519423]: Upload finished. err: <nil> Duration: 5.912261248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 30.291986372s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 07:59:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 30.29200206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X...X....XXX....X....X.XX..X....XX.XX..................X...X] duration: 30.004926816s trace: https://jaeger.dev.storj.io/trace/2d666678f6513905 FAIL (2025-12-30 07:59:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: 4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q) context deadline exceeded ======LOGS====== Dec 30 07:59:10.202653803 (592h30m10.073736848s) [upload=519423]: Starting upload Dec 30 07:59:10.202752560 (592h30m10.073835605s) [upload=519423]: Done starting upload Dec 30 07:59:10.202990363 (592h30m10.074073418s) [upload=519423]: Uploading segments... Dec 30 07:59:10.203134788 (592h30m10.074217833s) [upload=519423]: Got next segment. Inline: false Dec 30 07:59:10.203202307 (592h30m10.074285342s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 07:59:10.203208508 (592h30m10.074291553s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 07:59:10.447546036 (592h30m10.318629081s) [seg_pos={0 0}, upload=519423]: Begin upload segment... Dec 30 07:59:10.447642639 (592h30m10.318725684s) [seg_pos={0 0}, upload=519423]: Done begin upload segment. Dec 30 07:59:10.447706581 (592h30m10.318789626s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447713214 (592h30m10.318796249s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.447924818 (592h30m10.319007863s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448485695 (592h30m10.31956873s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.448958796 (592h30m10.320041841s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.449438610 (592h30m10.320521655s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450059773 (592h30m10.321142818s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.450651269 (592h30m10.321734314s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.452118853 (592h30m10.323201898s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453400123 (592h30m10.324483168s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.453661260 (592h30m10.324744305s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.455123524 (592h30m10.326206559s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.456465690 (592h30m10.327548726s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.458194782 (592h30m10.329277817s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.460128836 (592h30m10.331211881s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:10.462118644 (592h30m10.333201679s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Uploading piece... Dec 30 07:59:12.144433349 (592h30m12.015516395s) [upload=519423]: Next returned nil segment Dec 30 07:59:12.144446866 (592h30m12.015529911s) [upload=519423]: Waiting for error group managing segments... Dec 30 07:59:12.427594743 (592h30m12.298677788s) [noise=true , address=145.79.227.15:4874 , piece_id=6RMZQDJ7HHVFO6V5XWPCGBD6FSRKZ2T3A2YLT6BBCH7EWDKR4A4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.433067574 (592h30m12.304150619s) [noise=true , address=145.79.227.19:4079 , piece_id=5IR2PROF7XIVA67TT5RMKLOR3UEQ7HYMEQY2JQRW3WLZCRTC5A6Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.450044672 (592h30m12.321127717s) [noise=true , address=145.79.227.37:4566 , piece_id=W7NHLVHRKNOGTPIWFIQPTR7L6T622DFLC4PAPBRCOX5YNWAW26TQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.456286157 (592h30m12.327369192s) [noise=true , address=145.79.227.32:4836 , piece_id=6KX6MIJNAYODNKDASTTJ6AFYMGMW34IWAP34N26ZAIGJL7CUNR2A, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.460033563 (592h30m12.331116598s) [noise=true , address=145.79.227.47:4265 , piece_id=PUN4DB6GOVBALDAN6UUX3N5E2MW5VCZ3XLMGTJ5X33QWOMGBWS4Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.463362521 (592h30m12.334445556s) [noise=true , address=145.79.227.5:4222 , piece_id=ZZRFNLXE7QOLPT2SSUIEXHMYS45KYMZZDL4YFGQEV4WPFULHCIMA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.467905432 (592h30m12.338988467s) [noise=true , address=145.79.227.23:4546 , piece_id=GU5TCBM6LK54ZPVCAS5BL2B4IY4SE45W7ZM2CXM6H6QXIALYP2VQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.468015401 (592h30m12.339098446s) [noise=true , address=145.79.227.31:4798 , piece_id=WL2ASVHQ5K5LMVLCGC3XHM3P3TGIYRXM674QCSE3XEHDVOT7CLGA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.534010025 (592h30m12.405093061s) [noise=true , address=145.79.227.42:4122 , piece_id=ULNAKMSZLSMHR7TSEONHXV7MWBV7N7D4QWGPC2WWAPWWEXBGKJ3Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.553978358 (592h30m12.425061393s) [noise=true , address=145.79.227.7:4293 , piece_id=KAAXDITWS4XGVQHYLG6DP37BCRU65S6HVOG6QGBWPQVL4O2JZSCQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.653048103 (592h30m12.524131158s) [noise=true , address=77.90.31.19:4841 , piece_id=4WQX4J2O4JXAZI63LZPFJCZQTHBMJRQWVRC6ANM7DABRZNSDFZVA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:12.876260555 (592h30m12.7473436s) [noise=true , address=45.91.116.5:4297 , piece_id=6F3F2ACNPJVRN7BRKUD4BDKAZOH2BT2LRN24VNUJTXFPCT662VMQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265833353 (592h30m13.136916388s) [noise=true , address=77.90.31.10:4519 , piece_id=QDJTPOQ7WU6YRMN6W6RDZBFPRRRGLADSXONRQRBPT4HZYRLEWR2Q, seg_pos={0 0}, upload=519423]: Done uploading piece. err: <nil> Dec 30 07:59:13.265855445 (592h30m13.13693848s) [seg_pos={0 0}, upload=519423]: Segment reached optimal threshold of 13 pieces. Dec 30 07:59:13.266178440 (592h30m13.137261475s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: read tcp 188.166.225.170:46954->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 07:59:13.266190233 (592h30m13.137273308s) [noise=true , address=45.91.116.13:4155 , piece_id=JJOHYKDQK6BH3CJKJJVB7Z5DU52EOF7KVZQHZY3ICITAJWGLG3BQ, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266311203 (592h30m13.137394238s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266327554 (592h30m13.137410589s) [noise=true , address=77.90.31.13:4629 , piece_id=WWOEPNOL522R6M3SLDZZ65D5FJGT3MZXQN63FNDZVBSY4SMJAZNA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266376197 (592h30m13.137459232s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 07:59:13.266382398 (592h30m13.137465443s) [noise=true , address=45.91.116.8:4048 , piece_id=4YID6PJ53TFW7SEXQU6LMJIVBQ7NW4ZM52LSP4ITOUQD3NUALUBA, seg_pos={0 0}, upload=519423]: Long tail cancellation due to optimal threshold Dec 30 07:59:13.266397808 (592h30m13.137480843s) [seg_pos={0 0}, upload=519423]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 07:59:13.266510162 (592h30m13.137593207s) [upload=519423]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:59:13.266518638 (592h30m13.137601673s) [upload=519423]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:59:13.555405394 (592h30m13.426488439s) [upload=519423]: Done uploading segments... Dec 30 07:59:13.555425743 (592h30m13.426508778s) [upload=519423]: Upload finished. err: <nil> Duration: 30.004951031s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.....X........X...X...X.XX.XX.XXX.XXX.X.XXXXXX.XX.......X..] duration: 3.542754909s trace: https://jaeger.dev.storj.io/trace/2754cbfb05a23cc9 FAIL (2025-12-30 07:59:29 UTC) orbiter warning: context deadline exceeded Duration: 3.542903612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 37.411672174s trace: https://jaeger.dev.storj.io/trace/617a7cd0eceb7eee FAIL (2025-12-30 07:59:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.370921932 (592h25m21.242004977s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371427646 (592h25m21.242510691s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.371895877 (592h25m21.242978922s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372328671 (592h25m21.243411716s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.372730125 (592h25m21.24381316s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373094790 (592h25m21.244177825s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373435790 (592h25m21.244518835s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373763844 (592h25m21.244846889s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.373945009 (592h25m21.245028044s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374307490 (592h25m21.245390535s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.374714895 (592h25m21.24579794s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375312193 (592h25m21.246395228s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.375837673 (592h25m21.246920718s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376363535 (592h25m21.24744657s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:21.376883996 (592h25m21.247967051s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Uploading piece... Dec 30 07:54:22.754338978 (592h25m22.625422013s) [upload=519353]: Next returned nil segment Dec 30 07:54:22.754372011 (592h25m22.625455046s) [upload=519353]: Waiting for error group managing segments... Dec 30 07:54:23.133012562 (592h25m23.004095607s) [noise=true , address=64.49.13.138:4488 , piece_id=O7PTBN7PB5QPAJNCK7PQKQ3QM6HXVYFXIQTUYDYD65WCNLVSGSGQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.194533979 (592h25m23.065617014s) [noise=true , address=64.49.13.27:4966 , piece_id=ZHIE6NXMTC57Q4IGG57JVZRYK4L7BRAW6DWMBX2R7ZOEE35W67QQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.218062029 (592h25m23.089145065s) [noise=true , address=145.79.227.35:4485 , piece_id=A4PFMXSCTZSVXE4YB47J2CMWW2AK7LON6LN3EVWFLY7UFTXQOQ6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.228060909 (592h25m23.099143954s) [noise=true , address=145.79.227.7:4310 , piece_id=4GBL7QH5KFNFEXGNMZUPKUPIQCCIFU5IWS5Y4JBHGTB7CCI7YCQA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.239588670 (592h25m23.110671705s) [noise=true , address=145.79.227.43:4189 , piece_id=6GBVPGSZPSKCXVJQDYVGDEVUKA5WO626OSHRHGJMBGHBNCFDZJ7A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.252470899 (592h25m23.123553945s) [noise=true , address=145.79.227.11:4735 , piece_id=KQPQAEPQJ6VJFWJJGRELRQZBSABFX3PIUZIEV2JU27J2EHR633CA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.433283144 (592h25m23.304366189s) [noise=true , address=77.90.31.10:4549 , piece_id=RIFDVFU3DQPKBLUEO7SNE2U2HDD4GHK7Q47S2YFUJ4MNPZGVK4OA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.480802509 (592h25m23.351885544s) [noise=true , address=77.90.31.11:4582 , piece_id=27MMRCDSHOY3UCXDU5T7GHS4FCDB5WA632ZDPWC2OD7FHZRCZSAA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.592545475 (592h25m23.46362852s) [noise=true , address=145.79.227.33:4873 , piece_id=KKHXOJC7OKXI2L3ABVE3TRW4GVZG2BDCCJPRVOR2IQXBKPQ6HX5A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:23.662541078 (592h25m23.533624113s) [noise=true , address=45.91.116.12:4233 , piece_id=5MORTGRUM5BCJC3N2GY3ZUHCBAFMPIKW4UPVLH7HVWA44JRU63CQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.093012352 (592h25m23.964095388s) [noise=true , address=145.79.227.44:4226 , piece_id=U3QAO43EVDIRM7ITZHCRXBTZZS5EB3XLPZHKPIW2ESYSM3T5Y37A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.134687328 (592h25m24.005770373s) [noise=true , address=64.49.13.154:4936 , piece_id=TYVBCES3ANNYTZWZEP2ZEHUWUTHUGRR3QTIQZE37ZOU3IUKCGGTQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.279514141 (592h25m24.150597187s) [noise=true , address=145.79.227.34:4931 , piece_id=IW2WCS7F6P6OQM5V5HP23ICNVGTVRTKBKRZ4RZF3QLRYMG46CWEQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.299985662 (592h25m24.171068707s) [noise=true , address=145.79.227.30:4767 , piece_id=LLBCKYPJUKE2AK76NKCCXWLVZQBBK4GMCW7CPKWV5LGMSTGCRUYQ, seg_pos={0 0}, upload=519353]: Done uploading piece. err: <nil> Dec 30 07:54:24.300010499 (592h25m24.171093534s) [seg_pos={0 0}, upload=519353]: Segment reached optimal threshold of 14 pieces. Dec 30 07:54:24.300252610 (592h25m24.171335665s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38646->64.49.13.169:4705: use of closed network connection Dec 30 07:54:24.300262238 (592h25m24.171345294s) [noise=true , address=64.49.13.169:4705 , piece_id=ULNCOR3CH66IJPP7NQU7VVOGSNB5FQZOFWC4PIZ7U3WR6PV3LF7Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300435869 (592h25m24.171518914s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300458893 (592h25m24.171541928s) [noise=true , address=77.90.31.18:4820 , piece_id=LVC5RRCABELYAIYIR5LG4YZXQZ3CWVGIIKWHUQILEQSPN7JXGQHA, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300643474 (592h25m24.171726529s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ArtdfDCemaBzgtSYdVnuvLYUmKc4pgGLQEjmP1K1Wn4eoKTp4): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300654125 (592h25m24.17173717s) [noise=true , address=45.91.116.5:4308 , piece_id=GH6NHAW4WBVACD67UJWDMXCVJ6YPLMYA23CP5EHSYCNFDUHBNE6Q, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300749316 (592h25m24.171832451s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16aJgn61bhumVFmXfTv8YUi5JfZedtBqadv8yVcSEjgfgUxuDg): protocol: expected piece hash; context canceled; EOF Dec 30 07:54:24.300761400 (592h25m24.171844435s) [noise=true , address=145.79.227.49:4351 , piece_id=Q6CSWG7JHDJVKF757R2C2KU6UAZNAPCG67T6KGBMKZX5ZINJDH2A, seg_pos={0 0}, upload=519353]: Long tail cancellation due to optimal threshold Dec 30 07:54:24.300781517 (592h25m24.171864552s) [seg_pos={0 0}, upload=519353]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 07:54:24.300866640 (592h25m24.171949695s) [upload=519353]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 07:54:24.300873353 (592h25m24.171956398s) [upload=519353]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 07:54:24.598499370 (592h25m24.469582425s) [upload=519353]: Done uploading segments... Dec 30 07:54:24.598525330 (592h25m24.469608375s) [upload=519353]: Upload finished. err: <nil> Duration: 37.411692783s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)