Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X...........X...............XX........................] duration: 5.824581235s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:05:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.824606293s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.728241984s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:04:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.728295577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.613290958s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:04:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.61332353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........X...............XX.......................] duration: 5.843032332s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.843064082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X...........X...............XX......................] duration: 5.711445217s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:03:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.711471546s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.588832393s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:02:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588859835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X...........X...............XX.....................] duration: 5.678770961s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:02:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.678798764s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.730454006s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:01:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.730479905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 4.025367151s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:01:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.025399814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.........X...........X...............XX....................] duration: 5.99135213s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:01:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.991546921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.415345567s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:00:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.415400711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.........X...........X...............XX...................] duration: 6.011706447s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.011731505s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 4.865385278s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:59:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.865408763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.........X...........X...............XX..................] duration: 5.765173446s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.765221669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.........X...........X...............XX.................] duration: 5.923426475s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.923450791s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.424342998s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:58:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424368097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.405510427s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:58:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40565302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.........X...........X...............XX................] duration: 6.242879239s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.2429111s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 4.470727863s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:57:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.47075268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.430758276s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:57:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430785548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.........X...........X...............XX...............] duration: 5.672214878s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:56:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.672241218s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.454943376s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:56:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454978433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.........X...........X...............XX..............] duration: 5.763534214s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:55:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.763560525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.490428227s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:55:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490453264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.........X...........X...............XX.............] duration: 5.687600855s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.687774215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.........X...........X...............XX............] duration: 5.906169747s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.906198383s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.373423078s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:54:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.37345019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.........X...........X...............XX...........] duration: 5.921094206s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:53:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.921124895s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.59110866s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:53:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.591133727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.445774528s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:53:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445798764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.........X...........X...............XX..........] duration: 5.953528517s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:52:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.953553443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.........X...........X...............XX.........] duration: 5.628252406s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.628279897s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.621223102s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:51:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.621243131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.815504527s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:51:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.815535215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.........X...........X...............XX........] duration: 5.661163774s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:51:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.661194693s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.515634654s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:50:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515667728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.........X...........X...............XX.......] duration: 5.680451643s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:50:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.680505487s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.658334268s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:49:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.658362001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.........X...........X...............XX......] duration: 5.681987678s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:49:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.682012825s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.594864208s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:48:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.59488005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.........X...........X...............XX.....] duration: 6.203633469s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:48:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.203662585s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.646646312s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:48:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.646672612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X.........X...........X...............XX....] duration: 5.751472351s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:48:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.751501495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.577127696s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:47:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.577167021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X.........X...........X...............XX...] duration: 5.685901563s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:46:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.685930687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X.........X...........X...............XX..] duration: 5.758050759s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:46:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.758073162s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 4.516414547s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:46:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.516463541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X.........X...........X...............XX.] duration: 6.00928475s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:45:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.009302745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X.........X...........X...............XX] duration: 30.002630045s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:45:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 30.002653418s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 5.001540248s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:45:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00156238s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)