Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X...] duration: 5.816987971s trace: https://jaeger.dev.storj.io/trace/3382203165a15a75 FAIL (2025-12-27 15:25:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded ======LOGS====== 27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments... Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment... Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment. Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151 , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113 , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581 , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941 , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121 , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265 , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295 , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154 , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374 , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119 , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078 , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228 , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977 , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments... Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941 , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295 , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078 , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121 , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977 , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113 , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151 , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374 , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581 , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265 , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119 , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228 , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154 , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces. Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments... Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil> Duration: 5.81701393s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX.] duration: 5.001639602s trace: https://jaeger.dev.storj.io/trace/2209e5f196d4ee2a FAIL (2025-12-27 15:24:55 UTC) orbiter warning: context deadline exceeded Duration: 5.001666694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X..] duration: 5.993960568s trace: https://jaeger.dev.storj.io/trace/3382203165a15a75 FAIL (2025-12-27 15:24:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded ======LOGS====== 27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments... Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment... Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment. Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151 , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113 , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581 , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941 , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121 , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265 , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295 , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154 , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374 , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119 , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078 , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228 , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977 , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments... Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941 , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295 , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078 , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121 , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977 , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113 , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151 , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374 , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581 , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265 , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119 , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228 , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154 , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces. Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments... Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil> Duration: 5.993991897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X..............X....X...................X.....XX.] duration: 39.592708847s trace: https://jaeger.dev.storj.io/trace/1cf86c3db9e735d0 FAIL (2025-12-27 15:24:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.632070090 (527h53m34.503153125s) [noise=true , address=77.90.31.11:4565 , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.632523263 (527h53m34.503606298s) [noise=true , address=77.90.31.10:4540 , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.632676374 (527h53m34.503759419s) [noise=true , address=77.90.31.13:4659 , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.632931610 (527h53m34.504014655s) [noise=true , address=145.79.227.52:4466 , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.633357982 (527h53m34.504441017s) [noise=true , address=145.79.227.13:4816 , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.633490544 (527h53m34.504573579s) [noise=true , address=145.79.227.44:4218 , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.633891278 (527h53m34.504974313s) [noise=true , address=77.90.31.18:4820 , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.634043096 (527h53m34.505126141s) [noise=true , address=145.79.227.17:4997 , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.634325093 (527h53m34.505408128s) [noise=true , address=77.90.31.16:4739 , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.634722230 (527h53m34.505805265s) [noise=true , address=77.90.31.15:4716 , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.635114116 (527h53m34.506197151s) [noise=true , address=45.91.116.13:4171 , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.635719949 (527h53m34.506802984s) [noise=true , address=77.90.31.12:4618 , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.636184344 (527h53m34.507267379s) [noise=true , address=145.79.227.30:4775 , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:36.104663675 (527h53m35.97574671s) [upload=463022]: Next returned nil segment Dec 27 15:22:36.104675988 (527h53m35.975759023s) [upload=463022]: Waiting for error group managing segments... Dec 27 15:22:36.505615923 (527h53m36.376698958s) [noise=true , address=145.79.227.17:4997 , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.509752460 (527h53m36.380835505s) [noise=true , address=145.79.227.13:4816 , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.517493871 (527h53m36.388576906s) [noise=true , address=145.79.227.14:4865 , piece_id=XOXM2D6IS4SG6DN24DEGIPV7NK5FWDI7QVVJK7L3QB25AUPRNZIQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.623194766 (527h53m36.494277801s) [noise=true , address=145.79.227.52:4466 , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.749453706 (527h53m36.620536741s) [noise=true , address=45.91.116.11:4145 , piece_id=77YGIECBFASFKBHIQO333RHATPHZCA7JQW2WPNSGEGM24TTWDVGA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.780705355 (527h53m36.6517884s) [noise=true , address=77.90.31.18:4820 , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.803640327 (527h53m36.674723362s) [noise=true , address=77.90.31.13:4659 , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.841101831 (527h53m36.712184876s) [noise=true , address=77.90.31.16:4739 , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.481224175 (527h53m37.35230722s) [noise=true , address=77.90.31.15:4716 , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.504901952 (527h53m37.375984997s) [noise=true , address=145.79.227.30:4775 , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.553631302 (527h53m37.424714337s) [noise=true , address=145.79.227.44:4218 , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.596982441 (527h53m37.468065476s) [noise=true , address=145.79.227.40:4063 , piece_id=SRV3KMN3XXUOJA2GLEJ27UKWRANPEKCT3NC7BBDAVMTCI6FZCNPQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.792107735 (527h53m37.663190771s) [noise=true , address=77.90.31.17:4802 , piece_id=TFQZEPPQN5RYDUNTGLILJUHZ3QLTU7BT3QP67SS5Y4ZRUKPG6A5A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.802639390 (527h53m37.673722425s) [noise=true , address=77.90.31.12:4618 , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.802661622 (527h53m37.673744667s) [seg_pos={0 0}, upload=463022]: Segment reached optimal threshold of 14 pieces. Dec 27 15:22:37.802830423 (527h53m37.673913468s) [noise=true , address=77.90.31.11:4565 , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection Dec 27 15:22:37.802837557 (527h53m37.673920592s) [noise=true , address=77.90.31.11:4565 , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold Dec 27 15:22:37.803060181 (527h53m37.674143216s) [noise=true , address=45.91.116.13:4171 , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dmyMaZDYbCvoYvCc9m8Vf5dA8aJiw9qPh1G5AQWKnBig4mVg): protocol: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection Dec 27 15:22:37.803068377 (527h53m37.674151422s) [noise=true , address=45.91.116.13:4171 , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold Dec 27 15:22:37.803144622 (527h53m37.674227667s) [noise=true , address=64.49.13.169:4708 , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection Dec 27 15:22:37.803149080 (527h53m37.674232115s) [noise=true , address=64.49.13.169:4708 , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold Dec 27 15:22:37.803258279 (527h53m37.674341314s) [noise=true , address=77.90.31.10:4540 , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:37.803263358 (527h53m37.674346393s) [noise=true , address=77.90.31.10:4540 , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold Dec 27 15:22:37.803277855 (527h53m37.674360901s) [seg_pos={0 0}, upload=463022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:22:37.803341717 (527h53m37.674424762s) [upload=463022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:22:37.803346236 (527h53m37.674429271s) [upload=463022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:22:38.115241263 (527h53m37.986324298s) [upload=463022]: Done uploading segments... Dec 27 15:22:38.115265930 (527h53m37.986348975s) [upload=463022]: Upload finished. err: <nil> Duration: 39.592726571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX] duration: 5.00101679s trace: https://jaeger.dev.storj.io/trace/2209e5f196d4ee2a FAIL (2025-12-27 15:24:12 UTC) orbiter warning: context deadline exceeded Duration: 5.001039853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X.] duration: 6.404026519s trace: https://jaeger.dev.storj.io/trace/3382203165a15a75 FAIL (2025-12-27 15:23:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded ======LOGS====== 27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments... Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment... Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment. Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151 , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113 , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581 , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941 , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121 , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265 , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295 , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154 , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374 , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119 , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078 , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228 , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977 , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments... Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941 , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295 , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078 , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121 , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977 , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113 , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151 , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374 , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581 , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265 , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119 , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228 , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154 , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces. Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments... Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil> Duration: 6.404063189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...........X..............X....X...................X.....XX] duration: 1m0.002028377s trace: https://jaeger.dev.storj.io/trace/1cf86c3db9e735d0 FAIL (2025-12-27 15:23:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.632070090 (527h53m34.503153125s) [noise=true , address=77.90.31.11:4565 , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.632523263 (527h53m34.503606298s) [noise=true , address=77.90.31.10:4540 , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.632676374 (527h53m34.503759419s) [noise=true , address=77.90.31.13:4659 , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.632931610 (527h53m34.504014655s) [noise=true , address=145.79.227.52:4466 , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.633357982 (527h53m34.504441017s) [noise=true , address=145.79.227.13:4816 , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.633490544 (527h53m34.504573579s) [noise=true , address=145.79.227.44:4218 , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.633891278 (527h53m34.504974313s) [noise=true , address=77.90.31.18:4820 , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.634043096 (527h53m34.505126141s) [noise=true , address=145.79.227.17:4997 , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.634325093 (527h53m34.505408128s) [noise=true , address=77.90.31.16:4739 , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.634722230 (527h53m34.505805265s) [noise=true , address=77.90.31.15:4716 , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.635114116 (527h53m34.506197151s) [noise=true , address=45.91.116.13:4171 , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.635719949 (527h53m34.506802984s) [noise=true , address=77.90.31.12:4618 , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:34.636184344 (527h53m34.507267379s) [noise=true , address=145.79.227.30:4775 , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Uploading piece... Dec 27 15:22:36.104663675 (527h53m35.97574671s) [upload=463022]: Next returned nil segment Dec 27 15:22:36.104675988 (527h53m35.975759023s) [upload=463022]: Waiting for error group managing segments... Dec 27 15:22:36.505615923 (527h53m36.376698958s) [noise=true , address=145.79.227.17:4997 , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.509752460 (527h53m36.380835505s) [noise=true , address=145.79.227.13:4816 , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.517493871 (527h53m36.388576906s) [noise=true , address=145.79.227.14:4865 , piece_id=XOXM2D6IS4SG6DN24DEGIPV7NK5FWDI7QVVJK7L3QB25AUPRNZIQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.623194766 (527h53m36.494277801s) [noise=true , address=145.79.227.52:4466 , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.749453706 (527h53m36.620536741s) [noise=true , address=45.91.116.11:4145 , piece_id=77YGIECBFASFKBHIQO333RHATPHZCA7JQW2WPNSGEGM24TTWDVGA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.780705355 (527h53m36.6517884s) [noise=true , address=77.90.31.18:4820 , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.803640327 (527h53m36.674723362s) [noise=true , address=77.90.31.13:4659 , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:36.841101831 (527h53m36.712184876s) [noise=true , address=77.90.31.16:4739 , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.481224175 (527h53m37.35230722s) [noise=true , address=77.90.31.15:4716 , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.504901952 (527h53m37.375984997s) [noise=true , address=145.79.227.30:4775 , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.553631302 (527h53m37.424714337s) [noise=true , address=145.79.227.44:4218 , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.596982441 (527h53m37.468065476s) [noise=true , address=145.79.227.40:4063 , piece_id=SRV3KMN3XXUOJA2GLEJ27UKWRANPEKCT3NC7BBDAVMTCI6FZCNPQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.792107735 (527h53m37.663190771s) [noise=true , address=77.90.31.17:4802 , piece_id=TFQZEPPQN5RYDUNTGLILJUHZ3QLTU7BT3QP67SS5Y4ZRUKPG6A5A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.802639390 (527h53m37.673722425s) [noise=true , address=77.90.31.12:4618 , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil> Dec 27 15:22:37.802661622 (527h53m37.673744667s) [seg_pos={0 0}, upload=463022]: Segment reached optimal threshold of 14 pieces. Dec 27 15:22:37.802830423 (527h53m37.673913468s) [noise=true , address=77.90.31.11:4565 , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection Dec 27 15:22:37.802837557 (527h53m37.673920592s) [noise=true , address=77.90.31.11:4565 , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold Dec 27 15:22:37.803060181 (527h53m37.674143216s) [noise=true , address=45.91.116.13:4171 , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dmyMaZDYbCvoYvCc9m8Vf5dA8aJiw9qPh1G5AQWKnBig4mVg): protocol: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection Dec 27 15:22:37.803068377 (527h53m37.674151422s) [noise=true , address=45.91.116.13:4171 , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold Dec 27 15:22:37.803144622 (527h53m37.674227667s) [noise=true , address=64.49.13.169:4708 , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection Dec 27 15:22:37.803149080 (527h53m37.674232115s) [noise=true , address=64.49.13.169:4708 , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold Dec 27 15:22:37.803258279 (527h53m37.674341314s) [noise=true , address=77.90.31.10:4540 , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:37.803263358 (527h53m37.674346393s) [noise=true , address=77.90.31.10:4540 , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold Dec 27 15:22:37.803277855 (527h53m37.674360901s) [seg_pos={0 0}, upload=463022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:22:37.803341717 (527h53m37.674424762s) [upload=463022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:22:37.803346236 (527h53m37.674429271s) [upload=463022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:22:38.115241263 (527h53m37.986324298s) [upload=463022]: Done uploading segments... Dec 27 15:22:38.115265930 (527h53m37.986348975s) [upload=463022]: Upload finished. err: <nil> Duration: 1m0.002038956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X] duration: 30.003105037s trace: https://jaeger.dev.storj.io/trace/3382203165a15a75 FAIL (2025-12-27 15:23:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded ======LOGS====== 27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments... Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment... Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment. Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151 , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113 , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581 , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941 , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121 , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265 , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295 , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154 , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374 , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119 , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078 , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228 , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977 , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece... Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments... Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941 , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295 , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078 , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121 , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977 , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113 , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151 , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374 , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581 , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265 , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119 , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228 , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154 , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil> Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces. Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761 , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192 , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976 , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments... Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil> Duration: 30.003128943s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXX] duration: 5.001638028s trace: https://jaeger.dev.storj.io/trace/4d56275e57b7d1cf FAIL (2025-12-27 15:23:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001661423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...........X..............X....X...................X.....X] duration: 1m0.003716905s trace: https://jaeger.dev.storj.io/trace/3fabb9325ff74319 FAIL (2025-12-27 15:22:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4s) [noise=true , address=77.90.31.19:4860 , piece_id=ANNIF26M6O2O2QYF5BN5QEBR7SZPN52POARGWI7YTUQMCZVKDM5Q, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.369010277 (527h52m34.240093322s) [noise=true , address=77.90.31.12:4612 , piece_id=RNA453I2LXJ3H65AQ27TJDHMXPNBWGVY5YUPCBBBHBIQQVJDRNFQ, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.369404528 (527h52m34.240487563s) [noise=true , address=145.79.227.33:4882 , piece_id=ZQFLNXAGJJ2MQZSZVURDEX5OQAW3FUOMDMMLGF43UGZAVM2KAC6A, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.369911163 (527h52m34.240994208s) [noise=true , address=145.79.227.30:4768 , piece_id=QCA27RWVM2CZSQPX7YVHHTN5O2E7EFUY3GKXRASNDSHL6JFICPSA, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.370341542 (527h52m34.241424587s) [noise=true , address=45.91.116.14:4223 , piece_id=72TLZL42QCE6J465MGT4NK7Z3EKF5SNFEAPXC3DEOTC4ZJ3G75TQ, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.370759267 (527h52m34.241842302s) [noise=true , address=45.91.116.10:4113 , piece_id=FFWWG53B3H4LLFO6524OTXA3CQ5VFHMKUAL74IE4CYKGBC3X222A, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.371245233 (527h52m34.242328278s) [noise=true , address=77.90.31.13:4650 , piece_id=X5AECLLFHIQR2JASCHVBFN267DAWW3QTME2DWDZSL4GAHUXQ4TKQ, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.371737310 (527h52m34.242820355s) [noise=true , address=77.90.31.11:4566 , piece_id=5F32QOJPDB3UOIQDGLSG3YQYJEWMRGSMIZVXR2YJUTLGNMHUQ4MQ, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.372165416 (527h52m34.243248451s) [noise=true , address=145.79.227.34:4938 , piece_id=UTPXQZBMDY5SNGYWPH2I63LXFXQMM3SS5FC3LE6V4R5HZSH5QWXQ, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.372918970 (527h52m34.244002005s) [noise=true , address=77.90.31.21:4882 , piece_id=DMRHPTRACFA2A6E4JI2DNOOVLUZM7K43PQI4NH3QZUJ2QNGGW7SA, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.373404675 (527h52m34.24448771s) [noise=true , address=77.90.31.18:4825 , piece_id=SYRXD477QV743OHZC5PKQR3B5T5UEHB33BPAMWG6NPTGDUBTUGMQ, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.374011060 (527h52m34.245094105s) [noise=true , address=145.79.227.40:4078 , piece_id=TTY2JM753TXH3IEM57QHMAETKLQ7RK2B42TYUISVD62QUSAE3W2A, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.374503377 (527h52m34.245586422s) [noise=true , address=64.49.13.169:4706 , piece_id=BPFMA4R5AIHZNHZFG5O3WRPSH6MFIRZ2IKVSY6DONCIAMDUYHQRQ, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.374964655 (527h52m34.2460477s) [noise=true , address=77.90.31.10:4552 , piece_id=OON4JXAO6IRXWPKMGGWFDCHPTMVKOIZGAOM367MPJSJUAFD3OU4A, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.377921746 (527h52m34.249004791s) [noise=true , address=45.91.116.5:4329 , piece_id=JUPEFJMOGXJCGJGORVKODIAXEVEKC3CTRSFW4MJK7AEZYJX44A7Q, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:34.378561905 (527h52m34.24964495s) [noise=true , address=145.79.227.43:4192 , piece_id=FF4IWBETZLI6NKOXXFN5CTOJ3EXQC235HRU2XLKK7Q42WRE25J3A, seg_pos={0 0}, upload=463007]: Uploading piece... Dec 27 15:21:35.855010390 (527h52m35.726093425s) [upload=463007]: Next returned nil segment Dec 27 15:21:35.855021472 (527h52m35.726104517s) [upload=463007]: Waiting for error group managing segments... Dec 27 15:21:36.291926761 (527h52m36.163009806s) [noise=true , address=145.79.227.34:4938 , piece_id=UTPXQZBMDY5SNGYWPH2I63LXFXQMM3SS5FC3LE6V4R5HZSH5QWXQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:36.418807306 (527h52m36.289890341s) [noise=true , address=77.90.31.14:4686 , piece_id=ZVRJPH4ALTNQFI6OG3VB65LMBH4FNVMLB5N2RNVIREBFJ2ZOW6OQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:36.426031451 (527h52m36.297114496s) [noise=true , address=77.90.31.11:4566 , piece_id=5F32QOJPDB3UOIQDGLSG3YQYJEWMRGSMIZVXR2YJUTLGNMHUQ4MQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:36.706741176 (527h52m36.577824211s) [noise=true , address=45.91.116.10:4113 , piece_id=FFWWG53B3H4LLFO6524OTXA3CQ5VFHMKUAL74IE4CYKGBC3X222A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:36.707216270 (527h52m36.578299315s) [noise=true , address=77.90.31.18:4825 , piece_id=SYRXD477QV743OHZC5PKQR3B5T5UEHB33BPAMWG6NPTGDUBTUGMQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:36.751195264 (527h52m36.622278299s) [noise=true , address=45.91.116.14:4223 , piece_id=72TLZL42QCE6J465MGT4NK7Z3EKF5SNFEAPXC3DEOTC4ZJ3G75TQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:36.892739010 (527h52m36.763822045s) [noise=true , address=77.90.31.19:4860 , piece_id=ANNIF26M6O2O2QYF5BN5QEBR7SZPN52POARGWI7YTUQMCZVKDM5Q, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:36.905165182 (527h52m36.776248227s) [noise=true , address=77.90.31.13:4650 , piece_id=X5AECLLFHIQR2JASCHVBFN267DAWW3QTME2DWDZSL4GAHUXQ4TKQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:37.145642736 (527h52m37.016725771s) [noise=true , address=145.79.227.44:4208 , piece_id=FNRCXLIPXZV3T6CDNZUAY2BR3D6CAXQC4J2T5OKWVSQBXLTAAFAA, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:37.154877892 (527h52m37.025960927s) [noise=true , address=145.79.227.43:4192 , piece_id=FF4IWBETZLI6NKOXXFN5CTOJ3EXQC235HRU2XLKK7Q42WRE25J3A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:37.199396764 (527h52m37.070479799s) [noise=true , address=77.90.31.10:4552 , piece_id=OON4JXAO6IRXWPKMGGWFDCHPTMVKOIZGAOM367MPJSJUAFD3OU4A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:37.214836033 (527h52m37.085919078s) [noise=true , address=45.91.116.5:4329 , piece_id=JUPEFJMOGXJCGJGORVKODIAXEVEKC3CTRSFW4MJK7AEZYJX44A7Q, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:37.218432663 (527h52m37.089515718s) [noise=true , address=77.90.31.12:4612 , piece_id=RNA453I2LXJ3H65AQ27TJDHMXPNBWGVY5YUPCBBBHBIQQVJDRNFQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:37.265045552 (527h52m37.136128597s) [noise=true , address=145.79.227.30:4768 , piece_id=QCA27RWVM2CZSQPX7YVHHTN5O2E7EFUY3GKXRASNDSHL6JFICPSA, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil> Dec 27 15:21:37.265064668 (527h52m37.136147703s) [seg_pos={0 0}, upload=463007]: Segment reached optimal threshold of 14 pieces. Dec 27 15:21:37.265271973 (527h52m37.136355018s) [noise=true , address=145.79.227.40:4078 , piece_id=TTY2JM753TXH3IEM57QHMAETKLQ7RK2B42TYUISVD62QUSAE3W2A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 27 15:21:37.265280269 (527h52m37.136363314s) [noise=true , address=145.79.227.40:4078 , piece_id=TTY2JM753TXH3IEM57QHMAETKLQ7RK2B42TYUISVD62QUSAE3W2A, seg_pos={0 0}, upload=463007]: Long tail cancellation due to optimal threshold Dec 27 15:21:37.265551095 (527h52m37.13663414s) [noise=true , address=64.49.13.169:4706 , piece_id=BPFMA4R5AIHZNHZFG5O3WRPSH6MFIRZ2IKVSY6DONCIAMDUYHQRQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:55168->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:55168->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55168->64.49.13.169:4706: use of closed network connection Dec 27 15:21:37.265558649 (527h52m37.136641684s) [noise=true , address=64.49.13.169:4706 , piece_id=BPFMA4R5AIHZNHZFG5O3WRPSH6MFIRZ2IKVSY6DONCIAMDUYHQRQ, seg_pos={0 0}, upload=463007]: Long tail cancellation due to optimal threshold Dec 27 15:21:37.265788817 (527h52m37.136871862s) [noise=true , address=145.79.227.33:4882 , piece_id=ZQFLNXAGJJ2MQZSZVURDEX5OQAW3FUOMDMMLGF43UGZAVM2KAC6A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TU7roqQsKNGvNeqmWdVxMregk9Dqn3YZpfdvK6xp7c3zYvccc): protocol: expected piece hash; context canceled; EOF Dec 27 15:21:37.265848420 (527h52m37.136931485s) [noise=true , address=145.79.227.33:4882 , piece_id=ZQFLNXAGJJ2MQZSZVURDEX5OQAW3FUOMDMMLGF43UGZAVM2KAC6A, seg_pos={0 0}, upload=463007]: Long tail cancellation due to optimal threshold Dec 27 15:21:37.266011692 (527h52m37.137094727s) [noise=true , address=77.90.31.21:4882 , piece_id=DMRHPTRACFA2A6E4JI2DNOOVLUZM7K43PQI4NH3QZUJ2QNGGW7SA, seg_pos={0 0}, upload=463007]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 27 15:21:37.266019476 (527h52m37.137102511s) [noise=true , address=77.90.31.21:4882 , piece_id=DMRHPTRACFA2A6E4JI2DNOOVLUZM7K43PQI4NH3QZUJ2QNGGW7SA, seg_pos={0 0}, upload=463007]: Long tail cancellation due to optimal threshold Dec 27 15:21:37.266040246 (527h52m37.137123291s) [seg_pos={0 0}, upload=463007]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:21:37.266109889 (527h52m37.137192924s) [upload=463007]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:21:37.266115389 (527h52m37.137198424s) [upload=463007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:21:37.554391692 (527h52m37.425474747s) [upload=463007]: Done uploading segments... Dec 27 15:21:37.554419855 (527h52m37.4255029s) [upload=463007]: Upload finished. err: <nil> Duration: 1m0.003730651s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...] duration: 6.09157214s trace: https://jaeger.dev.storj.io/trace/4340d7129ab7e03f FAIL (2025-12-27 15:22:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ) context deadline exceeded ======LOGS====== Dec 27 15:19:35.326624967 (527h50m35.197708002s) [upload=462977]: Starting upload Dec 27 15:19:35.326698597 (527h50m35.197781642s) [upload=462977]: Done starting upload Dec 27 15:19:35.326775264 (527h50m35.197858299s) [upload=462977]: Uploading segments... Dec 27 15:19:35.326900632 (527h50m35.197983667s) [upload=462977]: Got next segment. Inline: false Dec 27 15:19:35.327029468 (527h50m35.198112513s) [upload=462977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:19:35.327035970 (527h50m35.198119015s) [upload=462977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:19:35.575151438 (527h50m35.446234493s) [seg_pos={0 0}, upload=462977]: Begin upload segment... Dec 27 15:19:35.575277387 (527h50m35.446360432s) [seg_pos={0 0}, upload=462977]: Done begin upload segment. Dec 27 15:19:35.575344425 (527h50m35.44642747s) [noise=true , address=145.79.227.35:4481 , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.575358512 (527h50m35.446441557s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.575431381 (527h50m35.446514436s) [noise=true , address=145.79.227.52:4447 , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576020322 (527h50m35.447103368s) [noise=true , address=45.91.116.13:4155 , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576457054 (527h50m35.447540089s) [noise=true , address=145.79.227.39:4015 , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576903584 (527h50m35.447986629s) [noise=true , address=45.91.116.7:4009 , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.577265433 (527h50m35.448348478s) [noise=true , address=45.91.116.10:4084 , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.577781526 (527h50m35.448864561s) [noise=true , address=145.79.227.48:4302 , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.578261690 (527h50m35.449344735s) [noise=true , address=145.79.227.16:4911 , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.578982232 (527h50m35.450065267s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.580552592 (527h50m35.451635638s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.580608849 (527h50m35.451691894s) [noise=true , address=77.90.31.11:4556 , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.582562860 (527h50m35.453645905s) [noise=true , address=77.90.31.10:4518 , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.584806623 (527h50m35.455889668s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.587466858 (527h50m35.458549913s) [noise=true , address=145.79.227.47:4265 , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.589399328 (527h50m35.460482373s) [noise=true , address=145.79.227.10:4403 , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:37.294264078 (527h50m37.165347113s) [upload=462977]: Next returned nil segment Dec 27 15:19:37.294277052 (527h50m37.165360087s) [upload=462977]: Waiting for error group managing segments... Dec 27 15:19:37.584769364 (527h50m37.455852399s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.587919692 (527h50m37.459002737s) [noise=true , address=145.79.227.48:4302 , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.595538479 (527h50m37.466621524s) [noise=true , address=145.79.227.10:4403 , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.597846745 (527h50m37.46892979s) [noise=true , address=145.79.227.39:4015 , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.601914871 (527h50m37.472997906s) [noise=true , address=145.79.227.35:4481 , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.610231108 (527h50m37.481314163s) [noise=true , address=145.79.227.16:4911 , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.610409387 (527h50m37.481492432s) [noise=true , address=145.79.227.47:4265 , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.611494423 (527h50m37.482577468s) [noise=true , address=145.79.227.52:4447 , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.802153223 (527h50m37.673236258s) [noise=true , address=45.91.116.7:4009 , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.847871228 (527h50m37.718954283s) [noise=true , address=77.90.31.11:4556 , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.919243645 (527h50m37.79032668s) [noise=true , address=45.91.116.10:4084 , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.986813229 (527h50m37.857896264s) [noise=true , address=45.91.116.13:4155 , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:38.458986455 (527h50m38.33006949s) [noise=true , address=77.90.31.10:4518 , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:38.459004680 (527h50m38.330087716s) [seg_pos={0 0}, upload=462977]: Segment reached optimal threshold of 13 pieces. Dec 27 15:19:38.459183651 (527h50m38.330266696s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459189492 (527h50m38.330272527s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459294222 (527h50m38.330377257s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459299892 (527h50m38.330382937s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459463304 (527h50m38.330546339s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459470407 (527h50m38.330553442s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459491307 (527h50m38.330574342s) [seg_pos={0 0}, upload=462977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:19:38.459555860 (527h50m38.330638895s) [upload=462977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:19:38.459561571 (527h50m38.330644606s) [upload=462977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:19:38.757600261 (527h50m38.628683316s) [upload=462977]: Done uploading segments... Dec 27 15:19:38.757625068 (527h50m38.628708123s) [upload=462977]: Upload finished. err: <nil> Duration: 6.09160436s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXX] duration: 5.001677465s trace: https://jaeger.dev.storj.io/trace/758320a971ae2408 FAIL (2025-12-27 15:22:06 UTC) orbiter warning: context deadline exceeded Duration: 5.001707421s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XX] duration: 5.001134057s trace: https://jaeger.dev.storj.io/trace/5ae7a6e60821a93b FAIL (2025-12-27 15:21:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001160007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...........X..............X....X...................X.....] duration: 43.172883275s trace: https://jaeger.dev.storj.io/trace/51d2e44a5388dd3e FAIL (2025-12-27 15:21:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, Piece ID: FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ) context deadline exceeded; piecestore: (Node ID: 12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, Piece ID: DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ) context deadline exceeded; piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ) context deadline exceeded ======LOGS====== , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887438882 (527h47m33.758521917s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887737339 (527h47m33.758820374s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.888853915 (527h47m33.75993696s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889312548 (527h47m33.760395594s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889894367 (527h47m33.760977412s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890793499 (527h47m33.761876544s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890936371 (527h47m33.762019416s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.892310177 (527h47m33.763393222s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.893865538 (527h47m33.764948583s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.897117241 (527h47m33.768200286s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.901733621 (527h47m33.772816666s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.902601734 (527h47m33.773684769s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.904727542 (527h47m33.775810578s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.905748506 (527h47m33.776831551s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:35.355522705 (527h47m35.22660574s) [upload=462935]: Next returned nil segment Dec 27 15:16:35.355540480 (527h47m35.226623515s) [upload=462935]: Waiting for error group managing segments... Dec 27 15:16:35.802401195 (527h47m35.67348423s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.963221887 (527h47m35.834304922s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.964808539 (527h47m35.835891584s) [noise=true , address=77.90.31.17:4798 , piece_id=Y7YSMCX3ZJUSIAXCWZ5UT6WULY7NDLEBT3DMVHAOZWQLLPUFQH3Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.976052790 (527h47m35.847135825s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.055389572 (527h47m35.926472617s) [noise=true , address=77.90.31.11:4583 , piece_id=KPBSYNDRBV7TMMDDGBLWPS53XNRTPVGCSVGV2EC4FY24XGWKRILQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.056167062 (527h47m35.927250097s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.088861078 (527h47m35.959944113s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.193327481 (527h47m36.064410516s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.650250267 (527h47m36.521333302s) [noise=true , address=145.79.227.44:4204 , piece_id=KAJQZK26B2SDCRPBZTSR2B3DWNGLNFCWAGXEWVX57CUKQYBDSQAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.692781133 (527h47m36.563864168s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.715571250 (527h47m36.586654285s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.889959848 (527h47m36.761042883s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.973000061 (527h47m36.844083096s) [noise=true , address=145.79.227.30:4769 , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295879234 (527h47m37.166962269s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295899252 (527h47m37.166982297s) [seg_pos={0 0}, upload=462935]: Segment reached optimal threshold of 14 pieces. Dec 27 15:16:37.296092550 (527h47m37.167175585s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296098371 (527h47m37.167181406s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296288694 (527h47m37.167371729s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296295597 (527h47m37.167378632s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296354058 (527h47m37.167437093s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection Dec 27 15:16:37.296371711 (527h47m37.167454746s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296451423 (527h47m37.167534458s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv): protocol: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection Dec 27 15:16:37.296457736 (527h47m37.167540781s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296482552 (527h47m37.167565587s) [seg_pos={0 0}, upload=462935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:16:37.296569268 (527h47m37.167652303s) [upload=462935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:16:37.296573766 (527h47m37.167656811s) [upload=462935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:16:37.598452278 (527h47m37.469535313s) [upload=462935]: Done uploading segments... Dec 27 15:16:37.598470573 (527h47m37.469553609s) [upload=462935]: Upload finished. err: <nil> Duration: 43.172906429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X..] duration: 5.975241392s trace: https://jaeger.dev.storj.io/trace/4340d7129ab7e03f FAIL (2025-12-27 15:21:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ) context deadline exceeded ======LOGS====== Dec 27 15:19:35.326624967 (527h50m35.197708002s) [upload=462977]: Starting upload Dec 27 15:19:35.326698597 (527h50m35.197781642s) [upload=462977]: Done starting upload Dec 27 15:19:35.326775264 (527h50m35.197858299s) [upload=462977]: Uploading segments... Dec 27 15:19:35.326900632 (527h50m35.197983667s) [upload=462977]: Got next segment. Inline: false Dec 27 15:19:35.327029468 (527h50m35.198112513s) [upload=462977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:19:35.327035970 (527h50m35.198119015s) [upload=462977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:19:35.575151438 (527h50m35.446234493s) [seg_pos={0 0}, upload=462977]: Begin upload segment... Dec 27 15:19:35.575277387 (527h50m35.446360432s) [seg_pos={0 0}, upload=462977]: Done begin upload segment. Dec 27 15:19:35.575344425 (527h50m35.44642747s) [noise=true , address=145.79.227.35:4481 , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.575358512 (527h50m35.446441557s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.575431381 (527h50m35.446514436s) [noise=true , address=145.79.227.52:4447 , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576020322 (527h50m35.447103368s) [noise=true , address=45.91.116.13:4155 , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576457054 (527h50m35.447540089s) [noise=true , address=145.79.227.39:4015 , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576903584 (527h50m35.447986629s) [noise=true , address=45.91.116.7:4009 , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.577265433 (527h50m35.448348478s) [noise=true , address=45.91.116.10:4084 , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.577781526 (527h50m35.448864561s) [noise=true , address=145.79.227.48:4302 , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.578261690 (527h50m35.449344735s) [noise=true , address=145.79.227.16:4911 , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.578982232 (527h50m35.450065267s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.580552592 (527h50m35.451635638s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.580608849 (527h50m35.451691894s) [noise=true , address=77.90.31.11:4556 , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.582562860 (527h50m35.453645905s) [noise=true , address=77.90.31.10:4518 , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.584806623 (527h50m35.455889668s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.587466858 (527h50m35.458549913s) [noise=true , address=145.79.227.47:4265 , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.589399328 (527h50m35.460482373s) [noise=true , address=145.79.227.10:4403 , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:37.294264078 (527h50m37.165347113s) [upload=462977]: Next returned nil segment Dec 27 15:19:37.294277052 (527h50m37.165360087s) [upload=462977]: Waiting for error group managing segments... Dec 27 15:19:37.584769364 (527h50m37.455852399s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.587919692 (527h50m37.459002737s) [noise=true , address=145.79.227.48:4302 , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.595538479 (527h50m37.466621524s) [noise=true , address=145.79.227.10:4403 , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.597846745 (527h50m37.46892979s) [noise=true , address=145.79.227.39:4015 , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.601914871 (527h50m37.472997906s) [noise=true , address=145.79.227.35:4481 , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.610231108 (527h50m37.481314163s) [noise=true , address=145.79.227.16:4911 , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.610409387 (527h50m37.481492432s) [noise=true , address=145.79.227.47:4265 , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.611494423 (527h50m37.482577468s) [noise=true , address=145.79.227.52:4447 , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.802153223 (527h50m37.673236258s) [noise=true , address=45.91.116.7:4009 , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.847871228 (527h50m37.718954283s) [noise=true , address=77.90.31.11:4556 , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.919243645 (527h50m37.79032668s) [noise=true , address=45.91.116.10:4084 , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.986813229 (527h50m37.857896264s) [noise=true , address=45.91.116.13:4155 , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:38.458986455 (527h50m38.33006949s) [noise=true , address=77.90.31.10:4518 , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:38.459004680 (527h50m38.330087716s) [seg_pos={0 0}, upload=462977]: Segment reached optimal threshold of 13 pieces. Dec 27 15:19:38.459183651 (527h50m38.330266696s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459189492 (527h50m38.330272527s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459294222 (527h50m38.330377257s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459299892 (527h50m38.330382937s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459463304 (527h50m38.330546339s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459470407 (527h50m38.330553442s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459491307 (527h50m38.330574342s) [seg_pos={0 0}, upload=462977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:19:38.459555860 (527h50m38.330638895s) [upload=462977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:19:38.459561571 (527h50m38.330644606s) [upload=462977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:19:38.757600261 (527h50m38.628683316s) [upload=462977]: Done uploading segments... Dec 27 15:19:38.757625068 (527h50m38.628708123s) [upload=462977]: Upload finished. err: <nil> Duration: 5.975271118s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.X] duration: 5.000871009s trace: https://jaeger.dev.storj.io/trace/29e2dcba2798532e FAIL (2025-12-27 15:21:01 UTC) orbiter warning: context deadline exceeded Duration: 5.000896316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X.] duration: 6.045625277s trace: https://jaeger.dev.storj.io/trace/4340d7129ab7e03f FAIL (2025-12-27 15:20:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ) context deadline exceeded ======LOGS====== Dec 27 15:19:35.326624967 (527h50m35.197708002s) [upload=462977]: Starting upload Dec 27 15:19:35.326698597 (527h50m35.197781642s) [upload=462977]: Done starting upload Dec 27 15:19:35.326775264 (527h50m35.197858299s) [upload=462977]: Uploading segments... Dec 27 15:19:35.326900632 (527h50m35.197983667s) [upload=462977]: Got next segment. Inline: false Dec 27 15:19:35.327029468 (527h50m35.198112513s) [upload=462977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:19:35.327035970 (527h50m35.198119015s) [upload=462977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:19:35.575151438 (527h50m35.446234493s) [seg_pos={0 0}, upload=462977]: Begin upload segment... Dec 27 15:19:35.575277387 (527h50m35.446360432s) [seg_pos={0 0}, upload=462977]: Done begin upload segment. Dec 27 15:19:35.575344425 (527h50m35.44642747s) [noise=true , address=145.79.227.35:4481 , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.575358512 (527h50m35.446441557s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.575431381 (527h50m35.446514436s) [noise=true , address=145.79.227.52:4447 , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576020322 (527h50m35.447103368s) [noise=true , address=45.91.116.13:4155 , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576457054 (527h50m35.447540089s) [noise=true , address=145.79.227.39:4015 , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576903584 (527h50m35.447986629s) [noise=true , address=45.91.116.7:4009 , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.577265433 (527h50m35.448348478s) [noise=true , address=45.91.116.10:4084 , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.577781526 (527h50m35.448864561s) [noise=true , address=145.79.227.48:4302 , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.578261690 (527h50m35.449344735s) [noise=true , address=145.79.227.16:4911 , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.578982232 (527h50m35.450065267s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.580552592 (527h50m35.451635638s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.580608849 (527h50m35.451691894s) [noise=true , address=77.90.31.11:4556 , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.582562860 (527h50m35.453645905s) [noise=true , address=77.90.31.10:4518 , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.584806623 (527h50m35.455889668s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.587466858 (527h50m35.458549913s) [noise=true , address=145.79.227.47:4265 , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.589399328 (527h50m35.460482373s) [noise=true , address=145.79.227.10:4403 , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:37.294264078 (527h50m37.165347113s) [upload=462977]: Next returned nil segment Dec 27 15:19:37.294277052 (527h50m37.165360087s) [upload=462977]: Waiting for error group managing segments... Dec 27 15:19:37.584769364 (527h50m37.455852399s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.587919692 (527h50m37.459002737s) [noise=true , address=145.79.227.48:4302 , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.595538479 (527h50m37.466621524s) [noise=true , address=145.79.227.10:4403 , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.597846745 (527h50m37.46892979s) [noise=true , address=145.79.227.39:4015 , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.601914871 (527h50m37.472997906s) [noise=true , address=145.79.227.35:4481 , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.610231108 (527h50m37.481314163s) [noise=true , address=145.79.227.16:4911 , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.610409387 (527h50m37.481492432s) [noise=true , address=145.79.227.47:4265 , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.611494423 (527h50m37.482577468s) [noise=true , address=145.79.227.52:4447 , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.802153223 (527h50m37.673236258s) [noise=true , address=45.91.116.7:4009 , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.847871228 (527h50m37.718954283s) [noise=true , address=77.90.31.11:4556 , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.919243645 (527h50m37.79032668s) [noise=true , address=45.91.116.10:4084 , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.986813229 (527h50m37.857896264s) [noise=true , address=45.91.116.13:4155 , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:38.458986455 (527h50m38.33006949s) [noise=true , address=77.90.31.10:4518 , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:38.459004680 (527h50m38.330087716s) [seg_pos={0 0}, upload=462977]: Segment reached optimal threshold of 13 pieces. Dec 27 15:19:38.459183651 (527h50m38.330266696s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459189492 (527h50m38.330272527s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459294222 (527h50m38.330377257s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459299892 (527h50m38.330382937s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459463304 (527h50m38.330546339s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459470407 (527h50m38.330553442s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459491307 (527h50m38.330574342s) [seg_pos={0 0}, upload=462977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:19:38.459555860 (527h50m38.330638895s) [upload=462977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:19:38.459561571 (527h50m38.330644606s) [upload=462977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:19:38.757600261 (527h50m38.628683316s) [upload=462977]: Done uploading segments... Dec 27 15:19:38.757625068 (527h50m38.628708123s) [upload=462977]: Upload finished. err: <nil> Duration: 6.045647007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X...........X..............X....X...................X....] duration: 47.93692753s trace: https://jaeger.dev.storj.io/trace/51d2e44a5388dd3e FAIL (2025-12-27 15:20:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, Piece ID: FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ) context deadline exceeded; piecestore: (Node ID: 12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, Piece ID: DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ) context deadline exceeded; piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ) context deadline exceeded ======LOGS====== , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887438882 (527h47m33.758521917s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887737339 (527h47m33.758820374s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.888853915 (527h47m33.75993696s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889312548 (527h47m33.760395594s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889894367 (527h47m33.760977412s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890793499 (527h47m33.761876544s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890936371 (527h47m33.762019416s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.892310177 (527h47m33.763393222s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.893865538 (527h47m33.764948583s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.897117241 (527h47m33.768200286s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.901733621 (527h47m33.772816666s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.902601734 (527h47m33.773684769s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.904727542 (527h47m33.775810578s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.905748506 (527h47m33.776831551s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:35.355522705 (527h47m35.22660574s) [upload=462935]: Next returned nil segment Dec 27 15:16:35.355540480 (527h47m35.226623515s) [upload=462935]: Waiting for error group managing segments... Dec 27 15:16:35.802401195 (527h47m35.67348423s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.963221887 (527h47m35.834304922s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.964808539 (527h47m35.835891584s) [noise=true , address=77.90.31.17:4798 , piece_id=Y7YSMCX3ZJUSIAXCWZ5UT6WULY7NDLEBT3DMVHAOZWQLLPUFQH3Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.976052790 (527h47m35.847135825s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.055389572 (527h47m35.926472617s) [noise=true , address=77.90.31.11:4583 , piece_id=KPBSYNDRBV7TMMDDGBLWPS53XNRTPVGCSVGV2EC4FY24XGWKRILQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.056167062 (527h47m35.927250097s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.088861078 (527h47m35.959944113s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.193327481 (527h47m36.064410516s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.650250267 (527h47m36.521333302s) [noise=true , address=145.79.227.44:4204 , piece_id=KAJQZK26B2SDCRPBZTSR2B3DWNGLNFCWAGXEWVX57CUKQYBDSQAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.692781133 (527h47m36.563864168s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.715571250 (527h47m36.586654285s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.889959848 (527h47m36.761042883s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.973000061 (527h47m36.844083096s) [noise=true , address=145.79.227.30:4769 , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295879234 (527h47m37.166962269s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295899252 (527h47m37.166982297s) [seg_pos={0 0}, upload=462935]: Segment reached optimal threshold of 14 pieces. Dec 27 15:16:37.296092550 (527h47m37.167175585s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296098371 (527h47m37.167181406s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296288694 (527h47m37.167371729s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296295597 (527h47m37.167378632s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296354058 (527h47m37.167437093s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection Dec 27 15:16:37.296371711 (527h47m37.167454746s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296451423 (527h47m37.167534458s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv): protocol: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection Dec 27 15:16:37.296457736 (527h47m37.167540781s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296482552 (527h47m37.167565587s) [seg_pos={0 0}, upload=462935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:16:37.296569268 (527h47m37.167652303s) [upload=462935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:16:37.296573766 (527h47m37.167656811s) [upload=462935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:16:37.598452278 (527h47m37.469535313s) [upload=462935]: Done uploading segments... Dec 27 15:16:37.598470573 (527h47m37.469553609s) [upload=462935]: Upload finished. err: <nil> Duration: 47.936942227s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.] duration: 5.002042237s trace: https://jaeger.dev.storj.io/trace/2fa9e2612d66081c FAIL (2025-12-27 15:20:37 UTC) orbiter warning: context deadline exceeded Duration: 5.002066553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X] duration: 30.005936528s trace: https://jaeger.dev.storj.io/trace/4340d7129ab7e03f FAIL (2025-12-27 15:20:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ) context deadline exceeded ======LOGS====== Dec 27 15:19:35.326624967 (527h50m35.197708002s) [upload=462977]: Starting upload Dec 27 15:19:35.326698597 (527h50m35.197781642s) [upload=462977]: Done starting upload Dec 27 15:19:35.326775264 (527h50m35.197858299s) [upload=462977]: Uploading segments... Dec 27 15:19:35.326900632 (527h50m35.197983667s) [upload=462977]: Got next segment. Inline: false Dec 27 15:19:35.327029468 (527h50m35.198112513s) [upload=462977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:19:35.327035970 (527h50m35.198119015s) [upload=462977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:19:35.575151438 (527h50m35.446234493s) [seg_pos={0 0}, upload=462977]: Begin upload segment... Dec 27 15:19:35.575277387 (527h50m35.446360432s) [seg_pos={0 0}, upload=462977]: Done begin upload segment. Dec 27 15:19:35.575344425 (527h50m35.44642747s) [noise=true , address=145.79.227.35:4481 , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.575358512 (527h50m35.446441557s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.575431381 (527h50m35.446514436s) [noise=true , address=145.79.227.52:4447 , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576020322 (527h50m35.447103368s) [noise=true , address=45.91.116.13:4155 , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576457054 (527h50m35.447540089s) [noise=true , address=145.79.227.39:4015 , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.576903584 (527h50m35.447986629s) [noise=true , address=45.91.116.7:4009 , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.577265433 (527h50m35.448348478s) [noise=true , address=45.91.116.10:4084 , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.577781526 (527h50m35.448864561s) [noise=true , address=145.79.227.48:4302 , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.578261690 (527h50m35.449344735s) [noise=true , address=145.79.227.16:4911 , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.578982232 (527h50m35.450065267s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.580552592 (527h50m35.451635638s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.580608849 (527h50m35.451691894s) [noise=true , address=77.90.31.11:4556 , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.582562860 (527h50m35.453645905s) [noise=true , address=77.90.31.10:4518 , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.584806623 (527h50m35.455889668s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.587466858 (527h50m35.458549913s) [noise=true , address=145.79.227.47:4265 , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:35.589399328 (527h50m35.460482373s) [noise=true , address=145.79.227.10:4403 , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Uploading piece... Dec 27 15:19:37.294264078 (527h50m37.165347113s) [upload=462977]: Next returned nil segment Dec 27 15:19:37.294277052 (527h50m37.165360087s) [upload=462977]: Waiting for error group managing segments... Dec 27 15:19:37.584769364 (527h50m37.455852399s) [noise=true , address=145.79.227.42:4121 , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.587919692 (527h50m37.459002737s) [noise=true , address=145.79.227.48:4302 , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.595538479 (527h50m37.466621524s) [noise=true , address=145.79.227.10:4403 , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.597846745 (527h50m37.46892979s) [noise=true , address=145.79.227.39:4015 , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.601914871 (527h50m37.472997906s) [noise=true , address=145.79.227.35:4481 , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.610231108 (527h50m37.481314163s) [noise=true , address=145.79.227.16:4911 , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.610409387 (527h50m37.481492432s) [noise=true , address=145.79.227.47:4265 , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.611494423 (527h50m37.482577468s) [noise=true , address=145.79.227.52:4447 , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.802153223 (527h50m37.673236258s) [noise=true , address=45.91.116.7:4009 , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.847871228 (527h50m37.718954283s) [noise=true , address=77.90.31.11:4556 , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.919243645 (527h50m37.79032668s) [noise=true , address=45.91.116.10:4084 , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:37.986813229 (527h50m37.857896264s) [noise=true , address=45.91.116.13:4155 , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:38.458986455 (527h50m38.33006949s) [noise=true , address=77.90.31.10:4518 , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil> Dec 27 15:19:38.459004680 (527h50m38.330087716s) [seg_pos={0 0}, upload=462977]: Segment reached optimal threshold of 13 pieces. Dec 27 15:19:38.459183651 (527h50m38.330266696s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459189492 (527h50m38.330272527s) [noise=true , address=145.79.227.30:4762 , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459294222 (527h50m38.330377257s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459299892 (527h50m38.330382937s) [noise=true , address=45.91.116.9:4261 , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459463304 (527h50m38.330546339s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 15:19:38.459470407 (527h50m38.330553442s) [noise=true , address=77.90.31.16:4736 , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold Dec 27 15:19:38.459491307 (527h50m38.330574342s) [seg_pos={0 0}, upload=462977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:19:38.459555860 (527h50m38.330638895s) [upload=462977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:19:38.459561571 (527h50m38.330644606s) [upload=462977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:19:38.757600261 (527h50m38.628683316s) [upload=462977]: Done uploading segments... Dec 27 15:19:38.757625068 (527h50m38.628708123s) [upload=462977]: Upload finished. err: <nil> Duration: 30.005956004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..X...........X..............X....X...................X...] duration: 50.93121874s trace: https://jaeger.dev.storj.io/trace/51d2e44a5388dd3e FAIL (2025-12-27 15:20:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, Piece ID: FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ) context deadline exceeded; piecestore: (Node ID: 12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, Piece ID: DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ) context deadline exceeded; piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ) context deadline exceeded ======LOGS====== , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887438882 (527h47m33.758521917s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887737339 (527h47m33.758820374s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.888853915 (527h47m33.75993696s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889312548 (527h47m33.760395594s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889894367 (527h47m33.760977412s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890793499 (527h47m33.761876544s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890936371 (527h47m33.762019416s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.892310177 (527h47m33.763393222s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.893865538 (527h47m33.764948583s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.897117241 (527h47m33.768200286s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.901733621 (527h47m33.772816666s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.902601734 (527h47m33.773684769s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.904727542 (527h47m33.775810578s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.905748506 (527h47m33.776831551s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:35.355522705 (527h47m35.22660574s) [upload=462935]: Next returned nil segment Dec 27 15:16:35.355540480 (527h47m35.226623515s) [upload=462935]: Waiting for error group managing segments... Dec 27 15:16:35.802401195 (527h47m35.67348423s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.963221887 (527h47m35.834304922s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.964808539 (527h47m35.835891584s) [noise=true , address=77.90.31.17:4798 , piece_id=Y7YSMCX3ZJUSIAXCWZ5UT6WULY7NDLEBT3DMVHAOZWQLLPUFQH3Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.976052790 (527h47m35.847135825s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.055389572 (527h47m35.926472617s) [noise=true , address=77.90.31.11:4583 , piece_id=KPBSYNDRBV7TMMDDGBLWPS53XNRTPVGCSVGV2EC4FY24XGWKRILQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.056167062 (527h47m35.927250097s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.088861078 (527h47m35.959944113s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.193327481 (527h47m36.064410516s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.650250267 (527h47m36.521333302s) [noise=true , address=145.79.227.44:4204 , piece_id=KAJQZK26B2SDCRPBZTSR2B3DWNGLNFCWAGXEWVX57CUKQYBDSQAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.692781133 (527h47m36.563864168s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.715571250 (527h47m36.586654285s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.889959848 (527h47m36.761042883s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.973000061 (527h47m36.844083096s) [noise=true , address=145.79.227.30:4769 , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295879234 (527h47m37.166962269s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295899252 (527h47m37.166982297s) [seg_pos={0 0}, upload=462935]: Segment reached optimal threshold of 14 pieces. Dec 27 15:16:37.296092550 (527h47m37.167175585s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296098371 (527h47m37.167181406s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296288694 (527h47m37.167371729s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296295597 (527h47m37.167378632s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296354058 (527h47m37.167437093s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection Dec 27 15:16:37.296371711 (527h47m37.167454746s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296451423 (527h47m37.167534458s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv): protocol: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection Dec 27 15:16:37.296457736 (527h47m37.167540781s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296482552 (527h47m37.167565587s) [seg_pos={0 0}, upload=462935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:16:37.296569268 (527h47m37.167652303s) [upload=462935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:16:37.296573766 (527h47m37.167656811s) [upload=462935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:16:37.598452278 (527h47m37.469535313s) [upload=462935]: Done uploading segments... Dec 27 15:16:37.598470573 (527h47m37.469553609s) [upload=462935]: Upload finished. err: <nil> Duration: 50.93123986s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX] duration: 5.001217325s trace: https://jaeger.dev.storj.io/trace/2fa9e2612d66081c FAIL (2025-12-27 15:19:27 UTC) orbiter warning: context deadline exceeded Duration: 5.001239276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..X...........X..............X....X...................X..] duration: 43.034516473s trace: https://jaeger.dev.storj.io/trace/51d2e44a5388dd3e FAIL (2025-12-27 15:19:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, Piece ID: FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ) context deadline exceeded; piecestore: (Node ID: 12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, Piece ID: DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ) context deadline exceeded; piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ) context deadline exceeded ======LOGS====== , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887438882 (527h47m33.758521917s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887737339 (527h47m33.758820374s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.888853915 (527h47m33.75993696s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889312548 (527h47m33.760395594s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889894367 (527h47m33.760977412s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890793499 (527h47m33.761876544s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890936371 (527h47m33.762019416s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.892310177 (527h47m33.763393222s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.893865538 (527h47m33.764948583s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.897117241 (527h47m33.768200286s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.901733621 (527h47m33.772816666s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.902601734 (527h47m33.773684769s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.904727542 (527h47m33.775810578s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.905748506 (527h47m33.776831551s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:35.355522705 (527h47m35.22660574s) [upload=462935]: Next returned nil segment Dec 27 15:16:35.355540480 (527h47m35.226623515s) [upload=462935]: Waiting for error group managing segments... Dec 27 15:16:35.802401195 (527h47m35.67348423s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.963221887 (527h47m35.834304922s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.964808539 (527h47m35.835891584s) [noise=true , address=77.90.31.17:4798 , piece_id=Y7YSMCX3ZJUSIAXCWZ5UT6WULY7NDLEBT3DMVHAOZWQLLPUFQH3Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.976052790 (527h47m35.847135825s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.055389572 (527h47m35.926472617s) [noise=true , address=77.90.31.11:4583 , piece_id=KPBSYNDRBV7TMMDDGBLWPS53XNRTPVGCSVGV2EC4FY24XGWKRILQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.056167062 (527h47m35.927250097s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.088861078 (527h47m35.959944113s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.193327481 (527h47m36.064410516s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.650250267 (527h47m36.521333302s) [noise=true , address=145.79.227.44:4204 , piece_id=KAJQZK26B2SDCRPBZTSR2B3DWNGLNFCWAGXEWVX57CUKQYBDSQAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.692781133 (527h47m36.563864168s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.715571250 (527h47m36.586654285s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.889959848 (527h47m36.761042883s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.973000061 (527h47m36.844083096s) [noise=true , address=145.79.227.30:4769 , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295879234 (527h47m37.166962269s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295899252 (527h47m37.166982297s) [seg_pos={0 0}, upload=462935]: Segment reached optimal threshold of 14 pieces. Dec 27 15:16:37.296092550 (527h47m37.167175585s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296098371 (527h47m37.167181406s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296288694 (527h47m37.167371729s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296295597 (527h47m37.167378632s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296354058 (527h47m37.167437093s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection Dec 27 15:16:37.296371711 (527h47m37.167454746s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296451423 (527h47m37.167534458s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv): protocol: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection Dec 27 15:16:37.296457736 (527h47m37.167540781s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296482552 (527h47m37.167565587s) [seg_pos={0 0}, upload=462935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:16:37.296569268 (527h47m37.167652303s) [upload=462935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:16:37.296573766 (527h47m37.167656811s) [upload=462935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:16:37.598452278 (527h47m37.469535313s) [upload=462935]: Done uploading segments... Dec 27 15:16:37.598470573 (527h47m37.469553609s) [upload=462935]: Upload finished. err: <nil> Duration: 43.034549125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....] duration: 5.791275933s trace: https://jaeger.dev.storj.io/trace/2509da23fbd2fcb5 FAIL (2025-12-27 15:18:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA) context deadline exceeded ======LOGS====== Dec 27 15:15:12.013603130 (527h46m11.884686185s) [upload=462916]: Starting upload Dec 27 15:15:12.013827617 (527h46m11.884910662s) [upload=462916]: Done starting upload Dec 27 15:15:12.014096839 (527h46m11.885179875s) [upload=462916]: Uploading segments... Dec 27 15:15:12.014148247 (527h46m11.885231292s) [upload=462916]: Got next segment. Inline: false Dec 27 15:15:12.014179056 (527h46m11.885262091s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:15:12.014184195 (527h46m11.88526724s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:15:12.261027251 (527h46m12.132110306s) [seg_pos={0 0}, upload=462916]: Begin upload segment... Dec 27 15:15:12.261137070 (527h46m12.132220105s) [seg_pos={0 0}, upload=462916]: Done begin upload segment. Dec 27 15:15:12.261230729 (527h46m12.132313774s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.261259083 (527h46m12.132342128s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262079955 (527h46m12.133163s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262357694 (527h46m12.133440739s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262798714 (527h46m12.133881749s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263197053 (527h46m12.134280099s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263619448 (527h46m12.134702483s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264020591 (527h46m12.135103636s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264412748 (527h46m12.135495793s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264838008 (527h46m12.135921053s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265229984 (527h46m12.136313019s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265551977 (527h46m12.136635022s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265646838 (527h46m12.136729873s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266195463 (527h46m12.137278508s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266583662 (527h46m12.137666707s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266995136 (527h46m12.138078181s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:13.981206883 (527h46m13.852289918s) [upload=462916]: Next returned nil segment Dec 27 15:15:13.981228043 (527h46m13.852311088s) [upload=462916]: Waiting for error group managing segments... Dec 27 15:15:14.258602335 (527h46m14.12968538s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.269337226 (527h46m14.140420271s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.276519823 (527h46m14.147602868s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.278562442 (527h46m14.149645497s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.284137208 (527h46m14.155220253s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.286468978 (527h46m14.157552013s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.287844237 (527h46m14.158927282s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290501486 (527h46m14.161584551s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290570668 (527h46m14.161653733s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.458489977 (527h46m14.329573022s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.523573930 (527h46m14.394656975s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.552996595 (527h46m14.424079631s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634796718 (527h46m14.505879763s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634983024 (527h46m14.506066069s) [seg_pos={0 0}, upload=462916]: Segment reached optimal threshold of 13 pieces. Dec 27 15:15:14.635218272 (527h46m14.506301317s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635227429 (527h46m14.506310474s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635390921 (527h46m14.506473976s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635411159 (527h46m14.506494204s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635519274 (527h46m14.506602329s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635527991 (527h46m14.506611036s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635579730 (527h46m14.506662775s) [seg_pos={0 0}, upload=462916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:15:14.635691673 (527h46m14.506774718s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:15:14.635698335 (527h46m14.50678138s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:15:14.940045619 (527h46m14.811128664s) [upload=462916]: Done uploading segments... Dec 27 15:15:14.940069575 (527h46m14.81115262s) [upload=462916]: Upload finished. err: <nil> Duration: 5.791314617s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXX] duration: 5.001267661s trace: https://jaeger.dev.storj.io/trace/309742bc094581bf FAIL (2025-12-27 15:18:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001286237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XX] duration: 5.001488442s trace: https://jaeger.dev.storj.io/trace/60c357aa038b8db9 FAIL (2025-12-27 15:18:14 UTC) orbiter warning: context deadline exceeded Duration: 5.001513188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X..X...........X..............X....X...................X.] duration: 40.579043561s trace: https://jaeger.dev.storj.io/trace/51d2e44a5388dd3e FAIL (2025-12-27 15:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, Piece ID: FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ) context deadline exceeded; piecestore: (Node ID: 12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, Piece ID: DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ) context deadline exceeded; piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ) context deadline exceeded ======LOGS====== , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887438882 (527h47m33.758521917s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887737339 (527h47m33.758820374s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.888853915 (527h47m33.75993696s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889312548 (527h47m33.760395594s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889894367 (527h47m33.760977412s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890793499 (527h47m33.761876544s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890936371 (527h47m33.762019416s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.892310177 (527h47m33.763393222s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.893865538 (527h47m33.764948583s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.897117241 (527h47m33.768200286s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.901733621 (527h47m33.772816666s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.902601734 (527h47m33.773684769s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.904727542 (527h47m33.775810578s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.905748506 (527h47m33.776831551s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:35.355522705 (527h47m35.22660574s) [upload=462935]: Next returned nil segment Dec 27 15:16:35.355540480 (527h47m35.226623515s) [upload=462935]: Waiting for error group managing segments... Dec 27 15:16:35.802401195 (527h47m35.67348423s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.963221887 (527h47m35.834304922s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.964808539 (527h47m35.835891584s) [noise=true , address=77.90.31.17:4798 , piece_id=Y7YSMCX3ZJUSIAXCWZ5UT6WULY7NDLEBT3DMVHAOZWQLLPUFQH3Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.976052790 (527h47m35.847135825s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.055389572 (527h47m35.926472617s) [noise=true , address=77.90.31.11:4583 , piece_id=KPBSYNDRBV7TMMDDGBLWPS53XNRTPVGCSVGV2EC4FY24XGWKRILQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.056167062 (527h47m35.927250097s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.088861078 (527h47m35.959944113s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.193327481 (527h47m36.064410516s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.650250267 (527h47m36.521333302s) [noise=true , address=145.79.227.44:4204 , piece_id=KAJQZK26B2SDCRPBZTSR2B3DWNGLNFCWAGXEWVX57CUKQYBDSQAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.692781133 (527h47m36.563864168s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.715571250 (527h47m36.586654285s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.889959848 (527h47m36.761042883s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.973000061 (527h47m36.844083096s) [noise=true , address=145.79.227.30:4769 , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295879234 (527h47m37.166962269s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295899252 (527h47m37.166982297s) [seg_pos={0 0}, upload=462935]: Segment reached optimal threshold of 14 pieces. Dec 27 15:16:37.296092550 (527h47m37.167175585s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296098371 (527h47m37.167181406s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296288694 (527h47m37.167371729s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296295597 (527h47m37.167378632s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296354058 (527h47m37.167437093s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection Dec 27 15:16:37.296371711 (527h47m37.167454746s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296451423 (527h47m37.167534458s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv): protocol: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection Dec 27 15:16:37.296457736 (527h47m37.167540781s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296482552 (527h47m37.167565587s) [seg_pos={0 0}, upload=462935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:16:37.296569268 (527h47m37.167652303s) [upload=462935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:16:37.296573766 (527h47m37.167656811s) [upload=462935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:16:37.598452278 (527h47m37.469535313s) [upload=462935]: Done uploading segments... Dec 27 15:16:37.598470573 (527h47m37.469553609s) [upload=462935]: Upload finished. err: <nil> Duration: 40.579061706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X....] duration: 5.950125201s trace: https://jaeger.dev.storj.io/trace/2509da23fbd2fcb5 FAIL (2025-12-27 15:18:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA) context deadline exceeded ======LOGS====== Dec 27 15:15:12.013603130 (527h46m11.884686185s) [upload=462916]: Starting upload Dec 27 15:15:12.013827617 (527h46m11.884910662s) [upload=462916]: Done starting upload Dec 27 15:15:12.014096839 (527h46m11.885179875s) [upload=462916]: Uploading segments... Dec 27 15:15:12.014148247 (527h46m11.885231292s) [upload=462916]: Got next segment. Inline: false Dec 27 15:15:12.014179056 (527h46m11.885262091s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:15:12.014184195 (527h46m11.88526724s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:15:12.261027251 (527h46m12.132110306s) [seg_pos={0 0}, upload=462916]: Begin upload segment... Dec 27 15:15:12.261137070 (527h46m12.132220105s) [seg_pos={0 0}, upload=462916]: Done begin upload segment. Dec 27 15:15:12.261230729 (527h46m12.132313774s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.261259083 (527h46m12.132342128s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262079955 (527h46m12.133163s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262357694 (527h46m12.133440739s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262798714 (527h46m12.133881749s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263197053 (527h46m12.134280099s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263619448 (527h46m12.134702483s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264020591 (527h46m12.135103636s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264412748 (527h46m12.135495793s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264838008 (527h46m12.135921053s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265229984 (527h46m12.136313019s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265551977 (527h46m12.136635022s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265646838 (527h46m12.136729873s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266195463 (527h46m12.137278508s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266583662 (527h46m12.137666707s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266995136 (527h46m12.138078181s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:13.981206883 (527h46m13.852289918s) [upload=462916]: Next returned nil segment Dec 27 15:15:13.981228043 (527h46m13.852311088s) [upload=462916]: Waiting for error group managing segments... Dec 27 15:15:14.258602335 (527h46m14.12968538s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.269337226 (527h46m14.140420271s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.276519823 (527h46m14.147602868s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.278562442 (527h46m14.149645497s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.284137208 (527h46m14.155220253s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.286468978 (527h46m14.157552013s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.287844237 (527h46m14.158927282s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290501486 (527h46m14.161584551s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290570668 (527h46m14.161653733s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.458489977 (527h46m14.329573022s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.523573930 (527h46m14.394656975s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.552996595 (527h46m14.424079631s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634796718 (527h46m14.505879763s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634983024 (527h46m14.506066069s) [seg_pos={0 0}, upload=462916]: Segment reached optimal threshold of 13 pieces. Dec 27 15:15:14.635218272 (527h46m14.506301317s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635227429 (527h46m14.506310474s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635390921 (527h46m14.506473976s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635411159 (527h46m14.506494204s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635519274 (527h46m14.506602329s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635527991 (527h46m14.506611036s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635579730 (527h46m14.506662775s) [seg_pos={0 0}, upload=462916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:15:14.635691673 (527h46m14.506774718s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:15:14.635698335 (527h46m14.50678138s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:15:14.940045619 (527h46m14.811128664s) [upload=462916]: Done uploading segments... Dec 27 15:15:14.940069575 (527h46m14.81115262s) [upload=462916]: Upload finished. err: <nil> Duration: 5.950150941s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...X] duration: 5.001491177s trace: https://jaeger.dev.storj.io/trace/148ab8ad480723e2 FAIL (2025-12-27 15:17:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001509521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X...] duration: 5.687635744s trace: https://jaeger.dev.storj.io/trace/2509da23fbd2fcb5 FAIL (2025-12-27 15:17:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA) context deadline exceeded ======LOGS====== Dec 27 15:15:12.013603130 (527h46m11.884686185s) [upload=462916]: Starting upload Dec 27 15:15:12.013827617 (527h46m11.884910662s) [upload=462916]: Done starting upload Dec 27 15:15:12.014096839 (527h46m11.885179875s) [upload=462916]: Uploading segments... Dec 27 15:15:12.014148247 (527h46m11.885231292s) [upload=462916]: Got next segment. Inline: false Dec 27 15:15:12.014179056 (527h46m11.885262091s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:15:12.014184195 (527h46m11.88526724s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:15:12.261027251 (527h46m12.132110306s) [seg_pos={0 0}, upload=462916]: Begin upload segment... Dec 27 15:15:12.261137070 (527h46m12.132220105s) [seg_pos={0 0}, upload=462916]: Done begin upload segment. Dec 27 15:15:12.261230729 (527h46m12.132313774s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.261259083 (527h46m12.132342128s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262079955 (527h46m12.133163s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262357694 (527h46m12.133440739s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262798714 (527h46m12.133881749s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263197053 (527h46m12.134280099s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263619448 (527h46m12.134702483s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264020591 (527h46m12.135103636s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264412748 (527h46m12.135495793s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264838008 (527h46m12.135921053s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265229984 (527h46m12.136313019s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265551977 (527h46m12.136635022s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265646838 (527h46m12.136729873s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266195463 (527h46m12.137278508s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266583662 (527h46m12.137666707s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266995136 (527h46m12.138078181s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:13.981206883 (527h46m13.852289918s) [upload=462916]: Next returned nil segment Dec 27 15:15:13.981228043 (527h46m13.852311088s) [upload=462916]: Waiting for error group managing segments... Dec 27 15:15:14.258602335 (527h46m14.12968538s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.269337226 (527h46m14.140420271s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.276519823 (527h46m14.147602868s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.278562442 (527h46m14.149645497s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.284137208 (527h46m14.155220253s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.286468978 (527h46m14.157552013s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.287844237 (527h46m14.158927282s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290501486 (527h46m14.161584551s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290570668 (527h46m14.161653733s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.458489977 (527h46m14.329573022s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.523573930 (527h46m14.394656975s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.552996595 (527h46m14.424079631s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634796718 (527h46m14.505879763s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634983024 (527h46m14.506066069s) [seg_pos={0 0}, upload=462916]: Segment reached optimal threshold of 13 pieces. Dec 27 15:15:14.635218272 (527h46m14.506301317s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635227429 (527h46m14.506310474s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635390921 (527h46m14.506473976s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635411159 (527h46m14.506494204s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635519274 (527h46m14.506602329s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635527991 (527h46m14.506611036s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635579730 (527h46m14.506662775s) [seg_pos={0 0}, upload=462916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:15:14.635691673 (527h46m14.506774718s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:15:14.635698335 (527h46m14.50678138s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:15:14.940045619 (527h46m14.811128664s) [upload=462916]: Done uploading segments... Dec 27 15:15:14.940069575 (527h46m14.81115262s) [upload=462916]: Upload finished. err: <nil> Duration: 5.687661472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X..X...........X..............X....X...................X] duration: 1m0.002876425s trace: https://jaeger.dev.storj.io/trace/51d2e44a5388dd3e FAIL (2025-12-27 15:17:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, Piece ID: FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ) context deadline exceeded; piecestore: (Node ID: 12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, Piece ID: DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ) context deadline exceeded; piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ) context deadline exceeded ======LOGS====== , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887438882 (527h47m33.758521917s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.887737339 (527h47m33.758820374s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.888853915 (527h47m33.75993696s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889312548 (527h47m33.760395594s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.889894367 (527h47m33.760977412s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890793499 (527h47m33.761876544s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.890936371 (527h47m33.762019416s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.892310177 (527h47m33.763393222s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.893865538 (527h47m33.764948583s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.897117241 (527h47m33.768200286s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.901733621 (527h47m33.772816666s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.902601734 (527h47m33.773684769s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.904727542 (527h47m33.775810578s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:33.905748506 (527h47m33.776831551s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Uploading piece... Dec 27 15:16:35.355522705 (527h47m35.22660574s) [upload=462935]: Next returned nil segment Dec 27 15:16:35.355540480 (527h47m35.226623515s) [upload=462935]: Waiting for error group managing segments... Dec 27 15:16:35.802401195 (527h47m35.67348423s) [noise=true , address=145.79.227.9:4373 , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.963221887 (527h47m35.834304922s) [noise=true , address=77.90.31.15:4711 , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.964808539 (527h47m35.835891584s) [noise=true , address=77.90.31.17:4798 , piece_id=Y7YSMCX3ZJUSIAXCWZ5UT6WULY7NDLEBT3DMVHAOZWQLLPUFQH3Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:35.976052790 (527h47m35.847135825s) [noise=true , address=77.90.31.16:4761 , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.055389572 (527h47m35.926472617s) [noise=true , address=77.90.31.11:4583 , piece_id=KPBSYNDRBV7TMMDDGBLWPS53XNRTPVGCSVGV2EC4FY24XGWKRILQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.056167062 (527h47m35.927250097s) [noise=true , address=77.90.31.10:4541 , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.088861078 (527h47m35.959944113s) [noise=true , address=45.91.116.9:4275 , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.193327481 (527h47m36.064410516s) [noise=true , address=45.91.116.13:4162 , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.650250267 (527h47m36.521333302s) [noise=true , address=145.79.227.44:4204 , piece_id=KAJQZK26B2SDCRPBZTSR2B3DWNGLNFCWAGXEWVX57CUKQYBDSQAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.692781133 (527h47m36.563864168s) [noise=true , address=145.79.227.40:4052 , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.715571250 (527h47m36.586654285s) [noise=true , address=145.79.227.33:4889 , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.889959848 (527h47m36.761042883s) [noise=true , address=77.90.31.18:4816 , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:36.973000061 (527h47m36.844083096s) [noise=true , address=145.79.227.30:4769 , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295879234 (527h47m37.166962269s) [noise=true , address=145.79.227.34:4919 , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil> Dec 27 15:16:37.295899252 (527h47m37.166982297s) [seg_pos={0 0}, upload=462935]: Segment reached optimal threshold of 14 pieces. Dec 27 15:16:37.296092550 (527h47m37.167175585s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296098371 (527h47m37.167181406s) [noise=true , address=77.90.31.13:4642 , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296288694 (527h47m37.167371729s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF Dec 27 15:16:37.296295597 (527h47m37.167378632s) [noise=true , address=77.90.31.21:4907 , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296354058 (527h47m37.167437093s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection Dec 27 15:16:37.296371711 (527h47m37.167454746s) [noise=true , address=64.49.13.169:4706 , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296451423 (527h47m37.167534458s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv): protocol: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection Dec 27 15:16:37.296457736 (527h47m37.167540781s) [noise=true , address=145.79.227.38:4984 , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold Dec 27 15:16:37.296482552 (527h47m37.167565587s) [seg_pos={0 0}, upload=462935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:16:37.296569268 (527h47m37.167652303s) [upload=462935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:16:37.296573766 (527h47m37.167656811s) [upload=462935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:16:37.598452278 (527h47m37.469535313s) [upload=462935]: Done uploading segments... Dec 27 15:16:37.598470573 (527h47m37.469553609s) [upload=462935]: Upload finished. err: <nil> Duration: 1m0.002887505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X..] duration: 5.871935285s trace: https://jaeger.dev.storj.io/trace/2509da23fbd2fcb5 FAIL (2025-12-27 15:17:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA) context deadline exceeded ======LOGS====== Dec 27 15:15:12.013603130 (527h46m11.884686185s) [upload=462916]: Starting upload Dec 27 15:15:12.013827617 (527h46m11.884910662s) [upload=462916]: Done starting upload Dec 27 15:15:12.014096839 (527h46m11.885179875s) [upload=462916]: Uploading segments... Dec 27 15:15:12.014148247 (527h46m11.885231292s) [upload=462916]: Got next segment. Inline: false Dec 27 15:15:12.014179056 (527h46m11.885262091s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:15:12.014184195 (527h46m11.88526724s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:15:12.261027251 (527h46m12.132110306s) [seg_pos={0 0}, upload=462916]: Begin upload segment... Dec 27 15:15:12.261137070 (527h46m12.132220105s) [seg_pos={0 0}, upload=462916]: Done begin upload segment. Dec 27 15:15:12.261230729 (527h46m12.132313774s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.261259083 (527h46m12.132342128s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262079955 (527h46m12.133163s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262357694 (527h46m12.133440739s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262798714 (527h46m12.133881749s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263197053 (527h46m12.134280099s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263619448 (527h46m12.134702483s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264020591 (527h46m12.135103636s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264412748 (527h46m12.135495793s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264838008 (527h46m12.135921053s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265229984 (527h46m12.136313019s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265551977 (527h46m12.136635022s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265646838 (527h46m12.136729873s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266195463 (527h46m12.137278508s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266583662 (527h46m12.137666707s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266995136 (527h46m12.138078181s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:13.981206883 (527h46m13.852289918s) [upload=462916]: Next returned nil segment Dec 27 15:15:13.981228043 (527h46m13.852311088s) [upload=462916]: Waiting for error group managing segments... Dec 27 15:15:14.258602335 (527h46m14.12968538s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.269337226 (527h46m14.140420271s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.276519823 (527h46m14.147602868s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.278562442 (527h46m14.149645497s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.284137208 (527h46m14.155220253s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.286468978 (527h46m14.157552013s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.287844237 (527h46m14.158927282s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290501486 (527h46m14.161584551s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290570668 (527h46m14.161653733s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.458489977 (527h46m14.329573022s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.523573930 (527h46m14.394656975s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.552996595 (527h46m14.424079631s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634796718 (527h46m14.505879763s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634983024 (527h46m14.506066069s) [seg_pos={0 0}, upload=462916]: Segment reached optimal threshold of 13 pieces. Dec 27 15:15:14.635218272 (527h46m14.506301317s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635227429 (527h46m14.506310474s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635390921 (527h46m14.506473976s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635411159 (527h46m14.506494204s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635519274 (527h46m14.506602329s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635527991 (527h46m14.506611036s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635579730 (527h46m14.506662775s) [seg_pos={0 0}, upload=462916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:15:14.635691673 (527h46m14.506774718s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:15:14.635698335 (527h46m14.50678138s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:15:14.940045619 (527h46m14.811128664s) [upload=462916]: Done uploading segments... Dec 27 15:15:14.940069575 (527h46m14.81115262s) [upload=462916]: Upload finished. err: <nil> Duration: 5.871962577s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...] duration: 5.001520574s trace: https://jaeger.dev.storj.io/trace/318c17fdd28b8d7c FAIL (2025-12-27 15:17:18 UTC) orbiter warning: context deadline exceeded Duration: 5.001549139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X...........X..............X....X...................] duration: 56.069739443s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:16:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 56.069765662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX..] duration: 5.001019798s trace: https://jaeger.dev.storj.io/trace/318c17fdd28b8d7c FAIL (2025-12-27 15:16:27 UTC) orbiter warning: context deadline exceeded Duration: 5.001046539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.] duration: 6.80631723s trace: https://jaeger.dev.storj.io/trace/2509da23fbd2fcb5 FAIL (2025-12-27 15:16:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA) context deadline exceeded ======LOGS====== Dec 27 15:15:12.013603130 (527h46m11.884686185s) [upload=462916]: Starting upload Dec 27 15:15:12.013827617 (527h46m11.884910662s) [upload=462916]: Done starting upload Dec 27 15:15:12.014096839 (527h46m11.885179875s) [upload=462916]: Uploading segments... Dec 27 15:15:12.014148247 (527h46m11.885231292s) [upload=462916]: Got next segment. Inline: false Dec 27 15:15:12.014179056 (527h46m11.885262091s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:15:12.014184195 (527h46m11.88526724s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:15:12.261027251 (527h46m12.132110306s) [seg_pos={0 0}, upload=462916]: Begin upload segment... Dec 27 15:15:12.261137070 (527h46m12.132220105s) [seg_pos={0 0}, upload=462916]: Done begin upload segment. Dec 27 15:15:12.261230729 (527h46m12.132313774s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.261259083 (527h46m12.132342128s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262079955 (527h46m12.133163s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262357694 (527h46m12.133440739s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262798714 (527h46m12.133881749s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263197053 (527h46m12.134280099s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263619448 (527h46m12.134702483s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264020591 (527h46m12.135103636s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264412748 (527h46m12.135495793s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264838008 (527h46m12.135921053s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265229984 (527h46m12.136313019s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265551977 (527h46m12.136635022s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265646838 (527h46m12.136729873s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266195463 (527h46m12.137278508s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266583662 (527h46m12.137666707s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266995136 (527h46m12.138078181s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:13.981206883 (527h46m13.852289918s) [upload=462916]: Next returned nil segment Dec 27 15:15:13.981228043 (527h46m13.852311088s) [upload=462916]: Waiting for error group managing segments... Dec 27 15:15:14.258602335 (527h46m14.12968538s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.269337226 (527h46m14.140420271s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.276519823 (527h46m14.147602868s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.278562442 (527h46m14.149645497s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.284137208 (527h46m14.155220253s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.286468978 (527h46m14.157552013s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.287844237 (527h46m14.158927282s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290501486 (527h46m14.161584551s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290570668 (527h46m14.161653733s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.458489977 (527h46m14.329573022s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.523573930 (527h46m14.394656975s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.552996595 (527h46m14.424079631s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634796718 (527h46m14.505879763s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634983024 (527h46m14.506066069s) [seg_pos={0 0}, upload=462916]: Segment reached optimal threshold of 13 pieces. Dec 27 15:15:14.635218272 (527h46m14.506301317s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635227429 (527h46m14.506310474s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635390921 (527h46m14.506473976s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635411159 (527h46m14.506494204s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635519274 (527h46m14.506602329s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635527991 (527h46m14.506611036s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635579730 (527h46m14.506662775s) [seg_pos={0 0}, upload=462916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:15:14.635691673 (527h46m14.506774718s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:15:14.635698335 (527h46m14.50678138s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:15:14.940045619 (527h46m14.811128664s) [upload=462916]: Done uploading segments... Dec 27 15:15:14.940069575 (527h46m14.81115262s) [upload=462916]: Upload finished. err: <nil> Duration: 6.806345334s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX.] duration: 4.324205038s trace: https://jaeger.dev.storj.io/trace/318c17fdd28b8d7c FAIL (2025-12-27 15:15:54 UTC) orbiter warning: context deadline exceeded Duration: 4.324235758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X] duration: 30.004556065s trace: https://jaeger.dev.storj.io/trace/2509da23fbd2fcb5 FAIL (2025-12-27 15:15:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA) context deadline exceeded ======LOGS====== Dec 27 15:15:12.013603130 (527h46m11.884686185s) [upload=462916]: Starting upload Dec 27 15:15:12.013827617 (527h46m11.884910662s) [upload=462916]: Done starting upload Dec 27 15:15:12.014096839 (527h46m11.885179875s) [upload=462916]: Uploading segments... Dec 27 15:15:12.014148247 (527h46m11.885231292s) [upload=462916]: Got next segment. Inline: false Dec 27 15:15:12.014179056 (527h46m11.885262091s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:15:12.014184195 (527h46m11.88526724s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:15:12.261027251 (527h46m12.132110306s) [seg_pos={0 0}, upload=462916]: Begin upload segment... Dec 27 15:15:12.261137070 (527h46m12.132220105s) [seg_pos={0 0}, upload=462916]: Done begin upload segment. Dec 27 15:15:12.261230729 (527h46m12.132313774s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.261259083 (527h46m12.132342128s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262079955 (527h46m12.133163s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262357694 (527h46m12.133440739s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.262798714 (527h46m12.133881749s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263197053 (527h46m12.134280099s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.263619448 (527h46m12.134702483s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264020591 (527h46m12.135103636s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264412748 (527h46m12.135495793s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.264838008 (527h46m12.135921053s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265229984 (527h46m12.136313019s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265551977 (527h46m12.136635022s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.265646838 (527h46m12.136729873s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266195463 (527h46m12.137278508s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266583662 (527h46m12.137666707s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:12.266995136 (527h46m12.138078181s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Uploading piece... Dec 27 15:15:13.981206883 (527h46m13.852289918s) [upload=462916]: Next returned nil segment Dec 27 15:15:13.981228043 (527h46m13.852311088s) [upload=462916]: Waiting for error group managing segments... Dec 27 15:15:14.258602335 (527h46m14.12968538s) [noise=true , address=145.79.227.29:4941 , piece_id=2N2SS4IXZABKNRA2KOMUS2WNS4UVDYVSBKNSK7WYTVSDQKTIOUPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.269337226 (527h46m14.140420271s) [noise=true , address=145.79.227.21:4475 , piece_id=EKVBJI4MMUATKXRQIOEV7R66VGUN4RWIDTHKZZFU5L2N4DQ4RVGQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.276519823 (527h46m14.147602868s) [noise=true , address=145.79.227.39:4013 , piece_id=LUMMFBJR65UFLDOCLPRPKSMZM4IVZVZD3CLQXE62GJFKQD3NQVMQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.278562442 (527h46m14.149645497s) [noise=true , address=145.79.227.51:4411 , piece_id=WXLRLJVY66CAQ4JDSQCJPRR6LEQIG3K2BK4FLJZVJBHAGBN3QLFQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.284137208 (527h46m14.155220253s) [noise=true , address=145.79.227.46:4602 , piece_id=UMKVHYZAEIVGSOCIKNKD5YLGXNWPA7XWM4ZWWPP2W6TBRASHYBZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.286468978 (527h46m14.157552013s) [noise=true , address=145.79.227.17:4972 , piece_id=N7R46OADATJQNRRLDQ4D3OYIRDFWVQHKFLFTTEWCGCCE35HULDHA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.287844237 (527h46m14.158927282s) [noise=true , address=145.79.227.3:4151 , piece_id=KQMXQAY4HGAANF23S6KFDASETE52VSVCDHPVROQMHYN4ECW37FVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290501486 (527h46m14.161584551s) [noise=true , address=145.79.227.4:4185 , piece_id=4D7CK7NFZUVY3Y6DQY3TGU77PC6KWJTKEDVOVFDMP5TC7E2BPC7Q, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.290570668 (527h46m14.161653733s) [noise=true , address=145.79.227.41:4086 , piece_id=3XI4JIH7NEDV5JWM3S4ZPTEN47SRHODRLAUTXR3ZEKJUNJLYTQLQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.458489977 (527h46m14.329573022s) [noise=true , address=77.90.31.16:4735 , piece_id=XIFSIUPFPAQ76TG3VIUDBLYKOR6S53BAV5LPHC6EIOVPMFI36YVA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.523573930 (527h46m14.394656975s) [noise=true , address=45.91.116.8:4046 , piece_id=C6URMRCVMHW4XIJ7KIFFTNG4W24L4YNCKZHXEVJ32UWPJMAMUQQA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.552996595 (527h46m14.424079631s) [noise=true , address=77.90.31.19:4844 , piece_id=UHEYADGDQJLODEUGIA6Z6STKMVJE62ZI4PYCI5RZ6LC5ERKCQPZA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634796718 (527h46m14.505879763s) [noise=true , address=45.91.116.4:4937 , piece_id=BOGZZRQZACYHQCGP3HZMYWZYJP3NH6XVL4XKNEOLH5M3KSL7DEYA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: <nil> Dec 27 15:15:14.634983024 (527h46m14.506066069s) [seg_pos={0 0}, upload=462916]: Segment reached optimal threshold of 13 pieces. Dec 27 15:15:14.635218272 (527h46m14.506301317s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635227429 (527h46m14.506310474s) [noise=true , address=45.91.116.13:4156 , piece_id=SLNAOULYNC35UP63PJTSUHVDISPSPGNFEGR3BICD5WBOEBAGTAPA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635390921 (527h46m14.506473976s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635411159 (527h46m14.506494204s) [noise=true , address=45.91.116.11:4120 , piece_id=OYUNNREKD5HS3DJLCLAFRCUMCM73VWAMSYIKYLIPN644YQKBIMSQ, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635519274 (527h46m14.506602329s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 15:15:14.635527991 (527h46m14.506611036s) [noise=true , address=45.91.116.14:4189 , piece_id=FMMRJT6YDPVLDW4MJULAFPUVXJVPN2XMXJ274KGXWRKEXRNJCLEA, seg_pos={0 0}, upload=462916]: Long tail cancellation due to optimal threshold Dec 27 15:15:14.635579730 (527h46m14.506662775s) [seg_pos={0 0}, upload=462916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:15:14.635691673 (527h46m14.506774718s) [upload=462916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:15:14.635698335 (527h46m14.50678138s) [upload=462916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:15:14.940045619 (527h46m14.811128664s) [upload=462916]: Done uploading segments... Dec 27 15:15:14.940069575 (527h46m14.81115262s) [upload=462916]: Upload finished. err: <nil> Duration: 30.004581393s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX] duration: 5.001509983s trace: https://jaeger.dev.storj.io/trace/318c17fdd28b8d7c FAIL (2025-12-27 15:15:29 UTC) orbiter warning: context deadline exceeded Duration: 5.001535493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..X...........X..............X....X..................] duration: 56.965648433s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 56.965663943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.] duration: 5.996114533s trace: https://jaeger.dev.storj.io/trace/d3b9ec460eb39eb FAIL (2025-12-27 15:14:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 15:13:56.503208831 (527h44m56.374291886s) [upload=462897]: Starting upload Dec 27 15:13:56.503301288 (527h44m56.374384343s) [upload=462897]: Done starting upload Dec 27 15:13:56.503399996 (527h44m56.374483041s) [upload=462897]: Uploading segments... Dec 27 15:13:56.503542096 (527h44m56.374625131s) [upload=462897]: Got next segment. Inline: false Dec 27 15:13:56.503623852 (527h44m56.374706907s) [upload=462897]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:13:56.503630474 (527h44m56.374713519s) [upload=462897]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:13:56.747613151 (527h44m56.618696187s) [seg_pos={0 0}, upload=462897]: Begin upload segment... Dec 27 15:13:56.747663747 (527h44m56.618746782s) [seg_pos={0 0}, upload=462897]: Done begin upload segment. Dec 27 15:13:56.747694005 (527h44m56.61877704s) [noise=true , address=145.79.227.44:4193 , piece_id=B7RM2LVOBHEZ2PEQ5TTQHPNLFKUJDN7PIN3UFMZ45SSOVTVVAF3A, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.752121657 (527h44m56.623204692s) [noise=true , address=45.91.116.5:4297 , piece_id=WPEEY5QHCKAARBTPBJSUP3C2K7FPYC22HBFWDS735NDDE4P4EKMQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.753368030 (527h44m56.624451065s) [noise=true , address=77.90.31.14:4662 , piece_id=WKD2TS63YAZINVRPYNGW3ED63F3QY32TCMSALI7BXFOTXHKXQKEQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.754516377 (527h44m56.625599422s) [noise=true , address=77.90.31.21:4880 , piece_id=EMGMAQVQUQ4OUW2KDG2BDCJUMKANSNQDGSSBPKVQ6OQSVPP3AOAQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.755745147 (527h44m56.626828182s) [noise=true , address=145.79.227.11:4732 , piece_id=SXYZKPRXJN24RECEUITTQCQXAPT3TRWCEAWLK2QDRWDIV5Q5KWPA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.756986200 (527h44m56.628069235s) [noise=true , address=145.79.227.14:4839 , piece_id=PN26CJE7VT3FMPN7RXARK7JUFOHLADSLWBV7NPXXTHQW767P4V4A, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.758156648 (527h44m56.629239693s) [noise=true , address=145.79.227.47:4265 , piece_id=DSW377AB5LCLZDXXRBYCERK7A5755LVQAIJR4LTBYEUTXUN4223A, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.759433209 (527h44m56.630516244s) [noise=true , address=145.79.227.39:4014 , piece_id=LT77JOGOHUGIDIBNBEI4AQ4Z77QXGZ42XF7OFLBEOETSZXMESGKQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.760688550 (527h44m56.631771595s) [noise=true , address=145.79.227.45:4229 , piece_id=C7CAXIQPN3DS65T72WKAMPIBPTB7SVIHSR4WHVRZN2RGNIBZTEFA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.762147778 (527h44m56.633230813s) [noise=true , address=45.91.116.11:4117 , piece_id=HZ3MOTUGNQLQZIKCIRINY2TEFKI3CSSGQD6TSXMMND54EGRN3IYQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.763366499 (527h44m56.634449544s) [noise=true , address=145.79.227.46:4602 , piece_id=F6ORQV7Y5BGCXBISMA3PNB2GJPJZWYKEI7PKMLO4XDRCC77U3GGA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.764677796 (527h44m56.635760841s) [noise=true , address=145.79.227.28:4726 , piece_id=2XBLNEIOOAINCIXB2XDUGSAWQQ2RJXBHWSIOTHOIDGZSKQUG2C2Q, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.765938036 (527h44m56.637021071s) [noise=true , address=77.90.31.11:4553 , piece_id=JYZ4MCGYWCLDSRYK5J5ABY6LWUYWYJXGY7UQYC72TSCU72JINFPQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.766782754 (527h44m56.637865799s) [noise=true , address=45.91.116.6:4974 , piece_id=TS7F3W4VAIXXKXBTPSZZD5NSFBEGS3BZKHCPO4CGDRWUGDLZRNKQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.768457643 (527h44m56.639540688s) [noise=true , address=145.79.227.38:4979 , piece_id=76EQBPXCUSJVSFZ2NB3FC3VZKDSYZ6WXLKC5ECSODDPXGC2LMHCA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.770143674 (527h44m56.641226709s) [noise=true , address=145.79.227.3:4150 , piece_id=HIKLJHAOPFO5WOJZ6ZSNYMYTAAPLQAS7PFAZ6ONQCBELHB24URHA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:58.486938960 (527h44m58.358021995s) [upload=462897]: Next returned nil segment Dec 27 15:13:58.486956793 (527h44m58.358039828s) [upload=462897]: Waiting for error group managing segments... Dec 27 15:13:58.771800609 (527h44m58.642883644s) [noise=true , address=145.79.227.11:4732 , piece_id=SXYZKPRXJN24RECEUITTQCQXAPT3TRWCEAWLK2QDRWDIV5Q5KWPA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.774366074 (527h44m58.645449109s) [noise=true , address=145.79.227.45:4229 , piece_id=C7CAXIQPN3DS65T72WKAMPIBPTB7SVIHSR4WHVRZN2RGNIBZTEFA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.774945948 (527h44m58.646028983s) [noise=true , address=145.79.227.46:4602 , piece_id=F6ORQV7Y5BGCXBISMA3PNB2GJPJZWYKEI7PKMLO4XDRCC77U3GGA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.777618097 (527h44m58.648701142s) [noise=true , address=145.79.227.39:4014 , piece_id=LT77JOGOHUGIDIBNBEI4AQ4Z77QXGZ42XF7OFLBEOETSZXMESGKQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.791993440 (527h44m58.663076485s) [noise=true , address=145.79.227.47:4265 , piece_id=DSW377AB5LCLZDXXRBYCERK7A5755LVQAIJR4LTBYEUTXUN4223A, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.792896710 (527h44m58.663979755s) [noise=true , address=145.79.227.28:4726 , piece_id=2XBLNEIOOAINCIXB2XDUGSAWQQ2RJXBHWSIOTHOIDGZSKQUG2C2Q, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.797615304 (527h44m58.66869834s) [noise=true , address=145.79.227.38:4979 , piece_id=76EQBPXCUSJVSFZ2NB3FC3VZKDSYZ6WXLKC5ECSODDPXGC2LMHCA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.798377205 (527h44m58.66946024s) [noise=true , address=145.79.227.3:4150 , piece_id=HIKLJHAOPFO5WOJZ6ZSNYMYTAAPLQAS7PFAZ6ONQCBELHB24URHA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:59.061086022 (527h44m58.932169077s) [noise=true , address=45.91.116.11:4117 , piece_id=HZ3MOTUGNQLQZIKCIRINY2TEFKI3CSSGQD6TSXMMND54EGRN3IYQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:59.202954877 (527h44m59.074037912s) [noise=true , address=45.91.116.5:4297 , piece_id=WPEEY5QHCKAARBTPBJSUP3C2K7FPYC22HBFWDS735NDDE4P4EKMQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:59.725329725 (527h44m59.59641276s) [noise=true , address=77.90.31.11:4553 , piece_id=JYZ4MCGYWCLDSRYK5J5ABY6LWUYWYJXGY7UQYC72TSCU72JINFPQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:59.831426273 (527h44m59.702509318s) [noise=true , address=77.90.31.21:4880 , piece_id=EMGMAQVQUQ4OUW2KDG2BDCJUMKANSNQDGSSBPKVQ6OQSVPP3AOAQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:14:00.018310414 (527h44m59.889393449s) [noise=true , address=145.79.227.44:4193 , piece_id=B7RM2LVOBHEZ2PEQ5TTQHPNLFKUJDN7PIN3UFMZ45SSOVTVVAF3A, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:14:00.018328519 (527h44m59.889411554s) [seg_pos={0 0}, upload=462897]: Segment reached optimal threshold of 13 pieces. Dec 27 15:14:00.018525755 (527h44m59.8896088s) [noise=true , address=145.79.227.14:4839 , piece_id=PN26CJE7VT3FMPN7RXARK7JUFOHLADSLWBV7NPXXTHQW767P4V4A, seg_pos={0 0}, upload=462897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Dec 27 15:14:00.018535092 (527h44m59.889618127s) [noise=true , address=145.79.227.14:4839 , piece_id=PN26CJE7VT3FMPN7RXARK7JUFOHLADSLWBV7NPXXTHQW767P4V4A, seg_pos={0 0}, upload=462897]: Long tail cancellation due to optimal threshold Dec 27 15:14:00.018553387 (527h44m59.889636422s) [noise=true , address=45.91.116.6:4974 , piece_id=TS7F3W4VAIXXKXBTPSZZD5NSFBEGS3BZKHCPO4CGDRWUGDLZRNKQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 15:14:00.018561373 (527h44m59.889644408s) [noise=true , address=45.91.116.6:4974 , piece_id=TS7F3W4VAIXXKXBTPSZZD5NSFBEGS3BZKHCPO4CGDRWUGDLZRNKQ, seg_pos={0 0}, upload=462897]: Long tail cancellation due to optimal threshold Dec 27 15:14:00.018761042 (527h44m59.889844087s) [noise=true , address=77.90.31.14:4662 , piece_id=WKD2TS63YAZINVRPYNGW3ED63F3QY32TCMSALI7BXFOTXHKXQKEQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 15:14:00.018772104 (527h44m59.889855159s) [noise=true , address=77.90.31.14:4662 , piece_id=WKD2TS63YAZINVRPYNGW3ED63F3QY32TCMSALI7BXFOTXHKXQKEQ, seg_pos={0 0}, upload=462897]: Long tail cancellation due to optimal threshold Dec 27 15:14:00.018792632 (527h44m59.889875667s) [seg_pos={0 0}, upload=462897]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:14:00.018873367 (527h44m59.889956412s) [upload=462897]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:14:00.018880230 (527h44m59.889963275s) [upload=462897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:14:00.316705103 (527h45m0.187788148s) [upload=462897]: Done uploading segments... Dec 27 15:14:00.316723018 (527h45m0.187806063s) [upload=462897]: Upload finished. err: <nil> Duration: 5.996130584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X...........X..............X....X.................] duration: 52.664864922s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:14:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 52.664886433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXX] duration: 5.00228646s trace: https://jaeger.dev.storj.io/trace/1642f414222b89e6 FAIL (2025-12-27 15:14:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00230684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X] duration: 30.005128574s trace: https://jaeger.dev.storj.io/trace/d3b9ec460eb39eb FAIL (2025-12-27 15:14:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 15:13:56.503208831 (527h44m56.374291886s) [upload=462897]: Starting upload Dec 27 15:13:56.503301288 (527h44m56.374384343s) [upload=462897]: Done starting upload Dec 27 15:13:56.503399996 (527h44m56.374483041s) [upload=462897]: Uploading segments... Dec 27 15:13:56.503542096 (527h44m56.374625131s) [upload=462897]: Got next segment. Inline: false Dec 27 15:13:56.503623852 (527h44m56.374706907s) [upload=462897]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:13:56.503630474 (527h44m56.374713519s) [upload=462897]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:13:56.747613151 (527h44m56.618696187s) [seg_pos={0 0}, upload=462897]: Begin upload segment... Dec 27 15:13:56.747663747 (527h44m56.618746782s) [seg_pos={0 0}, upload=462897]: Done begin upload segment. Dec 27 15:13:56.747694005 (527h44m56.61877704s) [noise=true , address=145.79.227.44:4193 , piece_id=B7RM2LVOBHEZ2PEQ5TTQHPNLFKUJDN7PIN3UFMZ45SSOVTVVAF3A, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.752121657 (527h44m56.623204692s) [noise=true , address=45.91.116.5:4297 , piece_id=WPEEY5QHCKAARBTPBJSUP3C2K7FPYC22HBFWDS735NDDE4P4EKMQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.753368030 (527h44m56.624451065s) [noise=true , address=77.90.31.14:4662 , piece_id=WKD2TS63YAZINVRPYNGW3ED63F3QY32TCMSALI7BXFOTXHKXQKEQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.754516377 (527h44m56.625599422s) [noise=true , address=77.90.31.21:4880 , piece_id=EMGMAQVQUQ4OUW2KDG2BDCJUMKANSNQDGSSBPKVQ6OQSVPP3AOAQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.755745147 (527h44m56.626828182s) [noise=true , address=145.79.227.11:4732 , piece_id=SXYZKPRXJN24RECEUITTQCQXAPT3TRWCEAWLK2QDRWDIV5Q5KWPA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.756986200 (527h44m56.628069235s) [noise=true , address=145.79.227.14:4839 , piece_id=PN26CJE7VT3FMPN7RXARK7JUFOHLADSLWBV7NPXXTHQW767P4V4A, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.758156648 (527h44m56.629239693s) [noise=true , address=145.79.227.47:4265 , piece_id=DSW377AB5LCLZDXXRBYCERK7A5755LVQAIJR4LTBYEUTXUN4223A, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.759433209 (527h44m56.630516244s) [noise=true , address=145.79.227.39:4014 , piece_id=LT77JOGOHUGIDIBNBEI4AQ4Z77QXGZ42XF7OFLBEOETSZXMESGKQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.760688550 (527h44m56.631771595s) [noise=true , address=145.79.227.45:4229 , piece_id=C7CAXIQPN3DS65T72WKAMPIBPTB7SVIHSR4WHVRZN2RGNIBZTEFA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.762147778 (527h44m56.633230813s) [noise=true , address=45.91.116.11:4117 , piece_id=HZ3MOTUGNQLQZIKCIRINY2TEFKI3CSSGQD6TSXMMND54EGRN3IYQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.763366499 (527h44m56.634449544s) [noise=true , address=145.79.227.46:4602 , piece_id=F6ORQV7Y5BGCXBISMA3PNB2GJPJZWYKEI7PKMLO4XDRCC77U3GGA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.764677796 (527h44m56.635760841s) [noise=true , address=145.79.227.28:4726 , piece_id=2XBLNEIOOAINCIXB2XDUGSAWQQ2RJXBHWSIOTHOIDGZSKQUG2C2Q, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.765938036 (527h44m56.637021071s) [noise=true , address=77.90.31.11:4553 , piece_id=JYZ4MCGYWCLDSRYK5J5ABY6LWUYWYJXGY7UQYC72TSCU72JINFPQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.766782754 (527h44m56.637865799s) [noise=true , address=45.91.116.6:4974 , piece_id=TS7F3W4VAIXXKXBTPSZZD5NSFBEGS3BZKHCPO4CGDRWUGDLZRNKQ, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.768457643 (527h44m56.639540688s) [noise=true , address=145.79.227.38:4979 , piece_id=76EQBPXCUSJVSFZ2NB3FC3VZKDSYZ6WXLKC5ECSODDPXGC2LMHCA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:56.770143674 (527h44m56.641226709s) [noise=true , address=145.79.227.3:4150 , piece_id=HIKLJHAOPFO5WOJZ6ZSNYMYTAAPLQAS7PFAZ6ONQCBELHB24URHA, seg_pos={0 0}, upload=462897]: Uploading piece... Dec 27 15:13:58.486938960 (527h44m58.358021995s) [upload=462897]: Next returned nil segment Dec 27 15:13:58.486956793 (527h44m58.358039828s) [upload=462897]: Waiting for error group managing segments... Dec 27 15:13:58.771800609 (527h44m58.642883644s) [noise=true , address=145.79.227.11:4732 , piece_id=SXYZKPRXJN24RECEUITTQCQXAPT3TRWCEAWLK2QDRWDIV5Q5KWPA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.774366074 (527h44m58.645449109s) [noise=true , address=145.79.227.45:4229 , piece_id=C7CAXIQPN3DS65T72WKAMPIBPTB7SVIHSR4WHVRZN2RGNIBZTEFA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.774945948 (527h44m58.646028983s) [noise=true , address=145.79.227.46:4602 , piece_id=F6ORQV7Y5BGCXBISMA3PNB2GJPJZWYKEI7PKMLO4XDRCC77U3GGA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.777618097 (527h44m58.648701142s) [noise=true , address=145.79.227.39:4014 , piece_id=LT77JOGOHUGIDIBNBEI4AQ4Z77QXGZ42XF7OFLBEOETSZXMESGKQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.791993440 (527h44m58.663076485s) [noise=true , address=145.79.227.47:4265 , piece_id=DSW377AB5LCLZDXXRBYCERK7A5755LVQAIJR4LTBYEUTXUN4223A, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.792896710 (527h44m58.663979755s) [noise=true , address=145.79.227.28:4726 , piece_id=2XBLNEIOOAINCIXB2XDUGSAWQQ2RJXBHWSIOTHOIDGZSKQUG2C2Q, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.797615304 (527h44m58.66869834s) [noise=true , address=145.79.227.38:4979 , piece_id=76EQBPXCUSJVSFZ2NB3FC3VZKDSYZ6WXLKC5ECSODDPXGC2LMHCA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:58.798377205 (527h44m58.66946024s) [noise=true , address=145.79.227.3:4150 , piece_id=HIKLJHAOPFO5WOJZ6ZSNYMYTAAPLQAS7PFAZ6ONQCBELHB24URHA, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:59.061086022 (527h44m58.932169077s) [noise=true , address=45.91.116.11:4117 , piece_id=HZ3MOTUGNQLQZIKCIRINY2TEFKI3CSSGQD6TSXMMND54EGRN3IYQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:59.202954877 (527h44m59.074037912s) [noise=true , address=45.91.116.5:4297 , piece_id=WPEEY5QHCKAARBTPBJSUP3C2K7FPYC22HBFWDS735NDDE4P4EKMQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:59.725329725 (527h44m59.59641276s) [noise=true , address=77.90.31.11:4553 , piece_id=JYZ4MCGYWCLDSRYK5J5ABY6LWUYWYJXGY7UQYC72TSCU72JINFPQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:13:59.831426273 (527h44m59.702509318s) [noise=true , address=77.90.31.21:4880 , piece_id=EMGMAQVQUQ4OUW2KDG2BDCJUMKANSNQDGSSBPKVQ6OQSVPP3AOAQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:14:00.018310414 (527h44m59.889393449s) [noise=true , address=145.79.227.44:4193 , piece_id=B7RM2LVOBHEZ2PEQ5TTQHPNLFKUJDN7PIN3UFMZ45SSOVTVVAF3A, seg_pos={0 0}, upload=462897]: Done uploading piece. err: <nil> Dec 27 15:14:00.018328519 (527h44m59.889411554s) [seg_pos={0 0}, upload=462897]: Segment reached optimal threshold of 13 pieces. Dec 27 15:14:00.018525755 (527h44m59.8896088s) [noise=true , address=145.79.227.14:4839 , piece_id=PN26CJE7VT3FMPN7RXARK7JUFOHLADSLWBV7NPXXTHQW767P4V4A, seg_pos={0 0}, upload=462897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Dec 27 15:14:00.018535092 (527h44m59.889618127s) [noise=true , address=145.79.227.14:4839 , piece_id=PN26CJE7VT3FMPN7RXARK7JUFOHLADSLWBV7NPXXTHQW767P4V4A, seg_pos={0 0}, upload=462897]: Long tail cancellation due to optimal threshold Dec 27 15:14:00.018553387 (527h44m59.889636422s) [noise=true , address=45.91.116.6:4974 , piece_id=TS7F3W4VAIXXKXBTPSZZD5NSFBEGS3BZKHCPO4CGDRWUGDLZRNKQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 15:14:00.018561373 (527h44m59.889644408s) [noise=true , address=45.91.116.6:4974 , piece_id=TS7F3W4VAIXXKXBTPSZZD5NSFBEGS3BZKHCPO4CGDRWUGDLZRNKQ, seg_pos={0 0}, upload=462897]: Long tail cancellation due to optimal threshold Dec 27 15:14:00.018761042 (527h44m59.889844087s) [noise=true , address=77.90.31.14:4662 , piece_id=WKD2TS63YAZINVRPYNGW3ED63F3QY32TCMSALI7BXFOTXHKXQKEQ, seg_pos={0 0}, upload=462897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 15:14:00.018772104 (527h44m59.889855159s) [noise=true , address=77.90.31.14:4662 , piece_id=WKD2TS63YAZINVRPYNGW3ED63F3QY32TCMSALI7BXFOTXHKXQKEQ, seg_pos={0 0}, upload=462897]: Long tail cancellation due to optimal threshold Dec 27 15:14:00.018792632 (527h44m59.889875667s) [seg_pos={0 0}, upload=462897]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:14:00.018873367 (527h44m59.889956412s) [upload=462897]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:14:00.018880230 (527h44m59.889963275s) [upload=462897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:14:00.316705103 (527h45m0.187788148s) [upload=462897]: Done uploading segments... Dec 27 15:14:00.316723018 (527h45m0.187806063s) [upload=462897]: Upload finished. err: <nil> Duration: 30.005155996s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XX] duration: 5.001898812s trace: https://jaeger.dev.storj.io/trace/5a837f6ef620ea67 FAIL (2025-12-27 15:13:39 UTC) orbiter warning: context deadline exceeded Duration: 5.001921096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..X...........X..............X....X................] duration: 46.353086445s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:13:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 46.35310409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..] duration: 6.093117819s trace: https://jaeger.dev.storj.io/trace/9ef47e79c76e3c FAIL (2025-12-27 15:13:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: GO26WT3V7MLTL3P6RJPZSMA6XSJJK3BAHNDHHFDUOGVTNCRIO56Q) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: I7OLBTDSHM2E3MT2NXO3EQJOJIJ42ZUR4S6RS27PHASVPPIDEBPQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PNSAHAOUROTNN6DNTWBJL4M72QT66X6DJYZYWBSIJAFJUMPXIS3A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: VDQIGD4MGI7UT3FS7BUHYDANGEXK44T2N5JGPQUSD7A5EGUJJ2SA) context deadline exceeded ======LOGS====== Dec 27 15:12:08.034433143 (527h43m7.905516178s) [upload=462866]: Starting upload Dec 27 15:12:08.034495652 (527h43m7.905578687s) [upload=462866]: Done starting upload Dec 27 15:12:08.034543863 (527h43m7.905626899s) [upload=462866]: Uploading segments... Dec 27 15:12:08.034679692 (527h43m7.905762727s) [upload=462866]: Got next segment. Inline: false Dec 27 15:12:08.034716232 (527h43m7.905799277s) [upload=462866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:12:08.034722043 (527h43m7.905805088s) [upload=462866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:12:08.280520358 (527h43m8.151603453s) [seg_pos={0 0}, upload=462866]: Begin upload segment... Dec 27 15:12:08.280588839 (527h43m8.151671874s) [seg_pos={0 0}, upload=462866]: Done begin upload segment. Dec 27 15:12:08.280645146 (527h43m8.151728181s) [noise=true , address=145.79.227.13:4804 , piece_id=BAF5JVNOEUTOFK25XD3Q6IM7IUH5SCPGFIRO3G7J2OMMXOKLJKKQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.280670394 (527h43m8.15175344s) [noise=true , address=45.91.116.7:4010 , piece_id=I7OLBTDSHM2E3MT2NXO3EQJOJIJ42ZUR4S6RS27PHASVPPIDEBPQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.281253113 (527h43m8.152336208s) [noise=true , address=77.90.31.15:4697 , piece_id=Q6RMVCVX2P5OR6VY5E4DK6NZGJTAKVJI7AARCRM7C5TGPOCU4XHQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.281853197 (527h43m8.152936242s) [noise=true , address=145.79.227.10:4402 , piece_id=64H6S6RCGZQX2JSTP6DZADEW6QUWJVTE2JCXWFHDKZKSBBU7QOEA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.282303454 (527h43m8.153386539s) [noise=true , address=145.79.227.34:4905 , piece_id=FJKSTIHRCE6AT7UP65NF474I52ZJ3DFJFWVWLSZINZSCGNBS42HA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.282741117 (527h43m8.153824152s) [noise=true , address=45.91.116.10:4082 , piece_id=PTQYXYN5LQVZFWWYCRNKPZXXOD74P6ASHWQWYFTQWG5FN6VD67CA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.283182798 (527h43m8.154265883s) [noise=true , address=145.79.227.35:4483 , piece_id=R6R3B7MD26KMUBPVGTDJ5BD46JVB3VZMTMMSMB3YVIYQAAQ5IFFA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.283687900 (527h43m8.154770945s) [noise=true , address=77.90.31.19:4841 , piece_id=PNSAHAOUROTNN6DNTWBJL4M72QT66X6DJYZYWBSIJAFJUMPXIS3A, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.284800198 (527h43m8.155883233s) [noise=true , address=145.79.227.16:4910 , piece_id=BD4WA2I4IXD7AJPFG5PLILKZQACSOQGPDLT5KAPQLO5OLJCC6GSA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.285256406 (527h43m8.156339451s) [noise=true , address=145.79.227.43:4157 , piece_id=QGL6S65NBMXVCFGZTX5XEZSGT3B6F56QIPEKMAIV6YA7CDB46LBA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.287140825 (527h43m8.15822387s) [noise=true , address=45.91.116.12:4226 , piece_id=VDQIGD4MGI7UT3FS7BUHYDANGEXK44T2N5JGPQUSD7A5EGUJJ2SA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.288834921 (527h43m8.159917956s) [noise=true , address=45.91.116.4:4937 , piece_id=VQIY64X57S3ILESNUDMF7VQGMNFFB5YOQFHCDU2PYP32D37QYALQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.290576116 (527h43m8.161659171s) [noise=true , address=45.91.116.9:4261 , piece_id=GO26WT3V7MLTL3P6RJPZSMA6XSJJK3BAHNDHHFDUOGVTNCRIO56Q, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.297361657 (527h43m8.168444692s) [noise=true , address=77.90.31.21:4878 , piece_id=2TALTBX2MPSYNS2QLNGEKJJHZOL7ORKZRF3P3KAB2CYZ265WNT7A, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.298123508 (527h43m8.169206553s) [noise=true , address=145.79.227.33:4869 , piece_id=IMN7KVDRWDLXCCXSW6TLHOFDD5TDU7TECKHLKG6A2UFEFZJBTVMQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.300661300 (527h43m8.171744335s) [noise=true , address=45.91.116.8:4046 , piece_id=TV3BPQVT7YGRHNGMPYRYP44SPGIA7ZOM5VY5HYYTCXWG4RQWG3VQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:10.019313594 (527h43m9.890396639s) [upload=462866]: Next returned nil segment Dec 27 15:12:10.019331457 (527h43m9.890414502s) [upload=462866]: Waiting for error group managing segments... Dec 27 15:12:10.299959074 (527h43m10.171042119s) [noise=true , address=145.79.227.13:4804 , piece_id=BAF5JVNOEUTOFK25XD3Q6IM7IUH5SCPGFIRO3G7J2OMMXOKLJKKQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.315374167 (527h43m10.186457212s) [noise=true , address=145.79.227.10:4402 , piece_id=64H6S6RCGZQX2JSTP6DZADEW6QUWJVTE2JCXWFHDKZKSBBU7QOEA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.319516635 (527h43m10.19059967s) [noise=true , address=145.79.227.16:4910 , piece_id=BD4WA2I4IXD7AJPFG5PLILKZQACSOQGPDLT5KAPQLO5OLJCC6GSA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.328698890 (527h43m10.199781925s) [noise=true , address=145.79.227.35:4483 , piece_id=R6R3B7MD26KMUBPVGTDJ5BD46JVB3VZMTMMSMB3YVIYQAAQ5IFFA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.483242871 (527h43m10.354325916s) [noise=true , address=77.90.31.19:4841 , piece_id=PNSAHAOUROTNN6DNTWBJL4M72QT66X6DJYZYWBSIJAFJUMPXIS3A, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.544807852 (527h43m10.415890897s) [noise=true , address=45.91.116.4:4937 , piece_id=VQIY64X57S3ILESNUDMF7VQGMNFFB5YOQFHCDU2PYP32D37QYALQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.570289603 (527h43m10.441372648s) [noise=true , address=45.91.116.9:4261 , piece_id=GO26WT3V7MLTL3P6RJPZSMA6XSJJK3BAHNDHHFDUOGVTNCRIO56Q, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.591930001 (527h43m10.463013036s) [noise=true , address=45.91.116.7:4010 , piece_id=I7OLBTDSHM2E3MT2NXO3EQJOJIJ42ZUR4S6RS27PHASVPPIDEBPQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.622874796 (527h43m10.493957841s) [noise=true , address=45.91.116.12:4226 , piece_id=VDQIGD4MGI7UT3FS7BUHYDANGEXK44T2N5JGPQUSD7A5EGUJJ2SA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.644503841 (527h43m10.515586876s) [noise=true , address=45.91.116.8:4046 , piece_id=TV3BPQVT7YGRHNGMPYRYP44SPGIA7ZOM5VY5HYYTCXWG4RQWG3VQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.725209519 (527h43m10.596292564s) [noise=true , address=77.90.31.15:4697 , piece_id=Q6RMVCVX2P5OR6VY5E4DK6NZGJTAKVJI7AARCRM7C5TGPOCU4XHQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:11.220157242 (527h43m11.091240277s) [noise=true , address=45.91.116.10:4082 , piece_id=PTQYXYN5LQVZFWWYCRNKPZXXOD74P6ASHWQWYFTQWG5FN6VD67CA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:11.474837471 (527h43m11.345920506s) [noise=true , address=145.79.227.43:4157 , piece_id=QGL6S65NBMXVCFGZTX5XEZSGT3B6F56QIPEKMAIV6YA7CDB46LBA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:11.474861656 (527h43m11.345944711s) [seg_pos={0 0}, upload=462866]: Segment reached optimal threshold of 13 pieces. Dec 27 15:12:11.475094520 (527h43m11.346177565s) [noise=true , address=145.79.227.33:4869 , piece_id=IMN7KVDRWDLXCCXSW6TLHOFDD5TDU7TECKHLKG6A2UFEFZJBTVMQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 15:12:11.475103066 (527h43m11.346186111s) [noise=true , address=145.79.227.33:4869 , piece_id=IMN7KVDRWDLXCCXSW6TLHOFDD5TDU7TECKHLKG6A2UFEFZJBTVMQ, seg_pos={0 0}, upload=462866]: Long tail cancellation due to optimal threshold Dec 27 15:12:11.475243233 (527h43m11.346326268s) [noise=true , address=77.90.31.21:4878 , piece_id=2TALTBX2MPSYNS2QLNGEKJJHZOL7ORKZRF3P3KAB2CYZ265WNT7A, seg_pos={0 0}, upload=462866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:55560->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:55560->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55560->77.90.31.21:4878: use of closed network connection Dec 27 15:12:11.475259194 (527h43m11.346342239s) [noise=true , address=77.90.31.21:4878 , piece_id=2TALTBX2MPSYNS2QLNGEKJJHZOL7ORKZRF3P3KAB2CYZ265WNT7A, seg_pos={0 0}, upload=462866]: Long tail cancellation due to optimal threshold Dec 27 15:12:11.475346149 (527h43m11.346429194s) [noise=true , address=145.79.227.34:4905 , piece_id=FJKSTIHRCE6AT7UP65NF474I52ZJ3DFJFWVWLSZINZSCGNBS42HA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 27 15:12:11.475354505 (527h43m11.34643754s) [noise=true , address=145.79.227.34:4905 , piece_id=FJKSTIHRCE6AT7UP65NF474I52ZJ3DFJFWVWLSZINZSCGNBS42HA, seg_pos={0 0}, upload=462866]: Long tail cancellation due to optimal threshold Dec 27 15:12:11.475377859 (527h43m11.346460904s) [seg_pos={0 0}, upload=462866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:12:11.475506284 (527h43m11.346589319s) [upload=462866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:12:11.475513868 (527h43m11.346596903s) [upload=462866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:12:11.774803511 (527h43m11.645886546s) [upload=462866]: Done uploading segments... Dec 27 15:12:11.774831605 (527h43m11.64591464s) [upload=462866]: Upload finished. err: <nil> Duration: 6.093143487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..X...........X..............X....X...............] duration: 28.719783698s trace: https://jaeger.dev.storj.io/trace/be41e8d6d19178d FAIL (2025-12-27 15:12:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 29m57.810517649s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.939788809 (527h29m57.810871844s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940060165 (527h29m57.81114321s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940352402 (527h29m57.811435437s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940617346 (527h29m57.811700381s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.940875418 (527h29m57.811958453s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941131345 (527h29m57.81221439s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.941443129 (527h29m57.812526164s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943144088 (527h29m57.814227123s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.943632468 (527h29m57.814715503s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.944845969 (527h29m57.815929014s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.946109354 (527h29m57.817192399s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.947349426 (527h29m57.818432461s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.948673357 (527h29m57.819756392s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.950134549 (527h29m57.821217584s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:57.951018853 (527h29m57.822101888s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Uploading piece... Dec 27 14:58:59.405479336 (527h29m59.276562381s) [upload=462676]: Next returned nil segment Dec 27 14:58:59.405495477 (527h29m59.276578522s) [upload=462676]: Waiting for error group managing segments... Dec 27 14:58:59.806909158 (527h29m59.677992193s) [noise=true , address=145.79.227.9:4384 , piece_id=4I4PQQQYFXEPXRJC5GUQ36LHDXZ4GENT4NDRKDNNEJ2XFWAQEP5A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.852751209 (527h29m59.723834254s) [noise=true , address=64.49.13.140:4513 , piece_id=XMQBKQ7C3XOAM7EWEDAJEHUFMFLBG7CAC55GA7YY4A57VWZA2CWQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.871257148 (527h29m59.742340183s) [noise=true , address=145.79.227.29:4975 , piece_id=XKE2KMKGSDV236JL2ETMMG3J7QZWFTRBAEYPBF7UHCGCHEVXWXOQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:58:59.993863782 (527h29m59.864946817s) [noise=true , address=77.90.31.19:4873 , piece_id=VEZN67KR2PIRD2PQHZH4LGSWQIW3BJEJ53EIAIUZWNO4DIZOJ3NQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.036971025 (527h29m59.90805406s) [noise=true , address=145.79.227.40:4063 , piece_id=5VHTQ5WWA3U3IVJTWP23PL5A73NFT4FPDRLM5RIBVXHQ3637F6XA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.106088897 (527h29m59.977171942s) [noise=true , address=77.90.31.16:4739 , piece_id=U634OBEWSZE2X4HLO7ZJELT2DKUPZRBUICJP6JNJP7EXUDODSTMQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.111446559 (527h29m59.982529634s) [noise=true , address=77.90.31.12:4617 , piece_id=G6AZXHTZOOJTYK4CQMWZ4AUEBDQ67PR3GCB5342DFRHGEN4ZRHPA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.143637787 (527h30m0.014720832s) [noise=true , address=77.90.31.17:4798 , piece_id=56IC3JS2T6JTSGEQZP6DX5PQX6WQW7QEHRYML7VVZSKI45I5NJ3A, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.173223904 (527h30m0.044306949s) [noise=true , address=77.90.31.18:4823 , piece_id=XPZCODUPWKCWCTGOYVJVQH72MADH4VDF4SCF2W5UVNQ75DW7LPIQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.280009453 (527h30m0.151092498s) [noise=true , address=45.91.116.13:4169 , piece_id=GXZIZAY3OHKFEBK3D6LJXU57WPZVAO7G3GWSOMFGQFJZXROMADGQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.718661972 (527h30m0.589745017s) [noise=true , address=145.79.227.44:4218 , piece_id=RI6WHAVKFDWTGKFGCJCQUQYO3V6LL3AWTCL3PQR24EXSJRLNGKOA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.759749620 (527h30m0.630832655s) [noise=true , address=145.79.227.34:4940 , piece_id=GXHCDTFPHBH4PTYAXZDQV6RTM3OM4SGKKJ7YJ6ZRA6PMMEFIL6CA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.796636398 (527h30m0.667719433s) [noise=true , address=145.79.227.30:4765 , piece_id=JPRYCAAB7566T4YXK6644PB6IXUIFVS62RMTYQVY6DOP7DGISANA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802939481 (527h30m0.674022517s) [noise=true , address=145.79.227.33:4886 , piece_id=DMY25YEHDVHPQSNRUFHCTPZKX2AXI4JCRBQL3XNCIKIJYZWUPZFA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: <nil> Dec 27 14:59:00.802967113 (527h30m0.674050158s) [seg_pos={0 0}, upload=462676]: Segment reached optimal threshold of 14 pieces. Dec 27 14:59:00.803195198 (527h30m0.674278233s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803206159 (527h30m0.674289204s) [noise=true , address=77.90.31.15:4701 , piece_id=Z5UNVDKY2QMX66KPEEZQBX66X7TRGL432T3DERUCK7X4ZW4QEFQQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803218162 (527h30m0.674301197s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44352->64.49.13.169:4708: use of closed network connection Dec 27 14:59:00.803226007 (527h30m0.674309052s) [noise=true , address=64.49.13.169:4708 , piece_id=PPPSGTIXFVDZL4XQYALDP6EFPMNV5Z6T3X3BP4ZPVO3GDTDLNG4Q, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803403254 (527h30m0.674486299s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803413403 (527h30m0.674496448s) [noise=true , address=77.90.31.14:4680 , piece_id=3ZXJVH7EZAHXGZWPLNR4LINCIY2NVHCYJCYECR7KEF66DBGHWRRQ, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803574030 (527h30m0.674657085s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Dec 27 14:59:00.803583507 (527h30m0.674666562s) [noise=true , address=77.90.31.13:4659 , piece_id=3TPA3MTKKANL52QNVDVIP6JLQYGQ6QJ4OTNPT7PPP6EBSPEAUCBA, seg_pos={0 0}, upload=462676]: Long tail cancellation due to optimal threshold Dec 27 14:59:00.803611882 (527h30m0.674694927s) [seg_pos={0 0}, upload=462676]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:59:00.803712744 (527h30m0.674795779s) [upload=462676]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:59:00.803720228 (527h30m0.674803263s) [upload=462676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:59:01.098516391 (527h30m0.969599446s) [upload=462676]: Done uploading segments... Dec 27 14:59:01.098538773 (527h30m0.969621818s) [upload=462676]: Upload finished. err: <nil> Duration: 28.719810499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXXXXXXXXXXX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.X] duration: 5.000816794s trace: https://jaeger.dev.storj.io/trace/76d2bb41de94405b FAIL (2025-12-27 15:12:44 UTC) orbiter warning: context deadline exceeded Duration: 5.0008409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X.X....X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X.] duration: 5.668374159s trace: https://jaeger.dev.storj.io/trace/9ef47e79c76e3c FAIL (2025-12-27 15:12:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: GO26WT3V7MLTL3P6RJPZSMA6XSJJK3BAHNDHHFDUOGVTNCRIO56Q) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: I7OLBTDSHM2E3MT2NXO3EQJOJIJ42ZUR4S6RS27PHASVPPIDEBPQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PNSAHAOUROTNN6DNTWBJL4M72QT66X6DJYZYWBSIJAFJUMPXIS3A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: VDQIGD4MGI7UT3FS7BUHYDANGEXK44T2N5JGPQUSD7A5EGUJJ2SA) context deadline exceeded ======LOGS====== Dec 27 15:12:08.034433143 (527h43m7.905516178s) [upload=462866]: Starting upload Dec 27 15:12:08.034495652 (527h43m7.905578687s) [upload=462866]: Done starting upload Dec 27 15:12:08.034543863 (527h43m7.905626899s) [upload=462866]: Uploading segments... Dec 27 15:12:08.034679692 (527h43m7.905762727s) [upload=462866]: Got next segment. Inline: false Dec 27 15:12:08.034716232 (527h43m7.905799277s) [upload=462866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:12:08.034722043 (527h43m7.905805088s) [upload=462866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:12:08.280520358 (527h43m8.151603453s) [seg_pos={0 0}, upload=462866]: Begin upload segment... Dec 27 15:12:08.280588839 (527h43m8.151671874s) [seg_pos={0 0}, upload=462866]: Done begin upload segment. Dec 27 15:12:08.280645146 (527h43m8.151728181s) [noise=true , address=145.79.227.13:4804 , piece_id=BAF5JVNOEUTOFK25XD3Q6IM7IUH5SCPGFIRO3G7J2OMMXOKLJKKQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.280670394 (527h43m8.15175344s) [noise=true , address=45.91.116.7:4010 , piece_id=I7OLBTDSHM2E3MT2NXO3EQJOJIJ42ZUR4S6RS27PHASVPPIDEBPQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.281253113 (527h43m8.152336208s) [noise=true , address=77.90.31.15:4697 , piece_id=Q6RMVCVX2P5OR6VY5E4DK6NZGJTAKVJI7AARCRM7C5TGPOCU4XHQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.281853197 (527h43m8.152936242s) [noise=true , address=145.79.227.10:4402 , piece_id=64H6S6RCGZQX2JSTP6DZADEW6QUWJVTE2JCXWFHDKZKSBBU7QOEA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.282303454 (527h43m8.153386539s) [noise=true , address=145.79.227.34:4905 , piece_id=FJKSTIHRCE6AT7UP65NF474I52ZJ3DFJFWVWLSZINZSCGNBS42HA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.282741117 (527h43m8.153824152s) [noise=true , address=45.91.116.10:4082 , piece_id=PTQYXYN5LQVZFWWYCRNKPZXXOD74P6ASHWQWYFTQWG5FN6VD67CA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.283182798 (527h43m8.154265883s) [noise=true , address=145.79.227.35:4483 , piece_id=R6R3B7MD26KMUBPVGTDJ5BD46JVB3VZMTMMSMB3YVIYQAAQ5IFFA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.283687900 (527h43m8.154770945s) [noise=true , address=77.90.31.19:4841 , piece_id=PNSAHAOUROTNN6DNTWBJL4M72QT66X6DJYZYWBSIJAFJUMPXIS3A, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.284800198 (527h43m8.155883233s) [noise=true , address=145.79.227.16:4910 , piece_id=BD4WA2I4IXD7AJPFG5PLILKZQACSOQGPDLT5KAPQLO5OLJCC6GSA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.285256406 (527h43m8.156339451s) [noise=true , address=145.79.227.43:4157 , piece_id=QGL6S65NBMXVCFGZTX5XEZSGT3B6F56QIPEKMAIV6YA7CDB46LBA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.287140825 (527h43m8.15822387s) [noise=true , address=45.91.116.12:4226 , piece_id=VDQIGD4MGI7UT3FS7BUHYDANGEXK44T2N5JGPQUSD7A5EGUJJ2SA, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.288834921 (527h43m8.159917956s) [noise=true , address=45.91.116.4:4937 , piece_id=VQIY64X57S3ILESNUDMF7VQGMNFFB5YOQFHCDU2PYP32D37QYALQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.290576116 (527h43m8.161659171s) [noise=true , address=45.91.116.9:4261 , piece_id=GO26WT3V7MLTL3P6RJPZSMA6XSJJK3BAHNDHHFDUOGVTNCRIO56Q, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.297361657 (527h43m8.168444692s) [noise=true , address=77.90.31.21:4878 , piece_id=2TALTBX2MPSYNS2QLNGEKJJHZOL7ORKZRF3P3KAB2CYZ265WNT7A, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.298123508 (527h43m8.169206553s) [noise=true , address=145.79.227.33:4869 , piece_id=IMN7KVDRWDLXCCXSW6TLHOFDD5TDU7TECKHLKG6A2UFEFZJBTVMQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:08.300661300 (527h43m8.171744335s) [noise=true , address=45.91.116.8:4046 , piece_id=TV3BPQVT7YGRHNGMPYRYP44SPGIA7ZOM5VY5HYYTCXWG4RQWG3VQ, seg_pos={0 0}, upload=462866]: Uploading piece... Dec 27 15:12:10.019313594 (527h43m9.890396639s) [upload=462866]: Next returned nil segment Dec 27 15:12:10.019331457 (527h43m9.890414502s) [upload=462866]: Waiting for error group managing segments... Dec 27 15:12:10.299959074 (527h43m10.171042119s) [noise=true , address=145.79.227.13:4804 , piece_id=BAF5JVNOEUTOFK25XD3Q6IM7IUH5SCPGFIRO3G7J2OMMXOKLJKKQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.315374167 (527h43m10.186457212s) [noise=true , address=145.79.227.10:4402 , piece_id=64H6S6RCGZQX2JSTP6DZADEW6QUWJVTE2JCXWFHDKZKSBBU7QOEA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.319516635 (527h43m10.19059967s) [noise=true , address=145.79.227.16:4910 , piece_id=BD4WA2I4IXD7AJPFG5PLILKZQACSOQGPDLT5KAPQLO5OLJCC6GSA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.328698890 (527h43m10.199781925s) [noise=true , address=145.79.227.35:4483 , piece_id=R6R3B7MD26KMUBPVGTDJ5BD46JVB3VZMTMMSMB3YVIYQAAQ5IFFA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.483242871 (527h43m10.354325916s) [noise=true , address=77.90.31.19:4841 , piece_id=PNSAHAOUROTNN6DNTWBJL4M72QT66X6DJYZYWBSIJAFJUMPXIS3A, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.544807852 (527h43m10.415890897s) [noise=true , address=45.91.116.4:4937 , piece_id=VQIY64X57S3ILESNUDMF7VQGMNFFB5YOQFHCDU2PYP32D37QYALQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.570289603 (527h43m10.441372648s) [noise=true , address=45.91.116.9:4261 , piece_id=GO26WT3V7MLTL3P6RJPZSMA6XSJJK3BAHNDHHFDUOGVTNCRIO56Q, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.591930001 (527h43m10.463013036s) [noise=true , address=45.91.116.7:4010 , piece_id=I7OLBTDSHM2E3MT2NXO3EQJOJIJ42ZUR4S6RS27PHASVPPIDEBPQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.622874796 (527h43m10.493957841s) [noise=true , address=45.91.116.12:4226 , piece_id=VDQIGD4MGI7UT3FS7BUHYDANGEXK44T2N5JGPQUSD7A5EGUJJ2SA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.644503841 (527h43m10.515586876s) [noise=true , address=45.91.116.8:4046 , piece_id=TV3BPQVT7YGRHNGMPYRYP44SPGIA7ZOM5VY5HYYTCXWG4RQWG3VQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:10.725209519 (527h43m10.596292564s) [noise=true , address=77.90.31.15:4697 , piece_id=Q6RMVCVX2P5OR6VY5E4DK6NZGJTAKVJI7AARCRM7C5TGPOCU4XHQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:11.220157242 (527h43m11.091240277s) [noise=true , address=45.91.116.10:4082 , piece_id=PTQYXYN5LQVZFWWYCRNKPZXXOD74P6ASHWQWYFTQWG5FN6VD67CA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:11.474837471 (527h43m11.345920506s) [noise=true , address=145.79.227.43:4157 , piece_id=QGL6S65NBMXVCFGZTX5XEZSGT3B6F56QIPEKMAIV6YA7CDB46LBA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: <nil> Dec 27 15:12:11.474861656 (527h43m11.345944711s) [seg_pos={0 0}, upload=462866]: Segment reached optimal threshold of 13 pieces. Dec 27 15:12:11.475094520 (527h43m11.346177565s) [noise=true , address=145.79.227.33:4869 , piece_id=IMN7KVDRWDLXCCXSW6TLHOFDD5TDU7TECKHLKG6A2UFEFZJBTVMQ, seg_pos={0 0}, upload=462866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 15:12:11.475103066 (527h43m11.346186111s) [noise=true , address=145.79.227.33:4869 , piece_id=IMN7KVDRWDLXCCXSW6TLHOFDD5TDU7TECKHLKG6A2UFEFZJBTVMQ, seg_pos={0 0}, upload=462866]: Long tail cancellation due to optimal threshold Dec 27 15:12:11.475243233 (527h43m11.346326268s) [noise=true , address=77.90.31.21:4878 , piece_id=2TALTBX2MPSYNS2QLNGEKJJHZOL7ORKZRF3P3KAB2CYZ265WNT7A, seg_pos={0 0}, upload=462866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:55560->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:55560->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55560->77.90.31.21:4878: use of closed network connection Dec 27 15:12:11.475259194 (527h43m11.346342239s) [noise=true , address=77.90.31.21:4878 , piece_id=2TALTBX2MPSYNS2QLNGEKJJHZOL7ORKZRF3P3KAB2CYZ265WNT7A, seg_pos={0 0}, upload=462866]: Long tail cancellation due to optimal threshold Dec 27 15:12:11.475346149 (527h43m11.346429194s) [noise=true , address=145.79.227.34:4905 , piece_id=FJKSTIHRCE6AT7UP65NF474I52ZJ3DFJFWVWLSZINZSCGNBS42HA, seg_pos={0 0}, upload=462866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 27 15:12:11.475354505 (527h43m11.34643754s) [noise=true , address=145.79.227.34:4905 , piece_id=FJKSTIHRCE6AT7UP65NF474I52ZJ3DFJFWVWLSZINZSCGNBS42HA, seg_pos={0 0}, upload=462866]: Long tail cancellation due to optimal threshold Dec 27 15:12:11.475377859 (527h43m11.346460904s) [seg_pos={0 0}, upload=462866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:12:11.475506284 (527h43m11.346589319s) [upload=462866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:12:11.475513868 (527h43m11.346596903s) [upload=462866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:12:11.774803511 (527h43m11.645886546s) [upload=462866]: Done uploading segments... Dec 27 15:12:11.774831605 (527h43m11.64591464s) [upload=462866]: Upload finished. err: <nil> Duration: 5.668389858s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)