Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.588832393s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:02:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588859835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X...........X...............XX.....................] duration: 5.678770961s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:02:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.678798764s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.730454006s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:01:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.730479905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 4.025367151s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:01:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.025399814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.........X...........X...............XX....................] duration: 5.99135213s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:01:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.991546921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.415345567s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:00:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.415400711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.........X...........X...............XX...................] duration: 6.011706447s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.011731505s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 4.865385278s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:59:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.865408763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.........X...........X...............XX..................] duration: 5.765173446s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.765221669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.........X...........X...............XX.................] duration: 5.923426475s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.923450791s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.424342998s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:58:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424368097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.405510427s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:58:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40565302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.........X...........X...............XX................] duration: 6.242879239s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.2429111s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 4.470727863s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:57:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.47075268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.430758276s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:57:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430785548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.........X...........X...............XX...............] duration: 5.672214878s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:56:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.672241218s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.454943376s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:56:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454978433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.........X...........X...............XX..............] duration: 5.763534214s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:55:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.763560525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.490428227s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:55:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490453264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.........X...........X...............XX.............] duration: 5.687600855s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.687774215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.........X...........X...............XX............] duration: 5.906169747s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.906198383s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.373423078s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:54:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.37345019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.........X...........X...............XX...........] duration: 5.921094206s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:53:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.921124895s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.59110866s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:53:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.591133727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.445774528s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:53:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445798764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.........X...........X...............XX..........] duration: 5.953528517s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:52:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.953553443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.........X...........X...............XX.........] duration: 5.628252406s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.628279897s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.621223102s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:51:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.621243131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.815504527s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:51:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.815535215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.........X...........X...............XX........] duration: 5.661163774s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:51:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.661194693s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.515634654s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:50:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515667728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.........X...........X...............XX.......] duration: 5.680451643s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:50:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.680505487s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.658334268s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:49:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.658362001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.........X...........X...............XX......] duration: 5.681987678s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:49:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.682012825s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.594864208s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:48:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.59488005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.........X...........X...............XX.....] duration: 6.203633469s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:48:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.203662585s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.646646312s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:48:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.646672612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X.........X...........X...............XX....] duration: 5.751472351s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:48:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.751501495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.577127696s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:47:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.577167021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X.........X...........X...............XX...] duration: 5.685901563s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:46:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.685930687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X.........X...........X...............XX..] duration: 5.758050759s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:46:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.758073162s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 4.516414547s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:46:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.516463541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X.........X...........X...............XX.] duration: 6.00928475s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:45:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.009302745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X.........X...........X...............XX] duration: 30.002630045s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:45:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 30.002653418s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 5.001540248s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:45:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00156238s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.416064976s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:44:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416092399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.728899062s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:44:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.728925603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.........X...........X...............X] duration: 30.003399582s trace: https://jaeger.dev.storj.io/trace/3eff27cd9e59e739 FAIL (2025-12-25 09:44:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: HFA4RRO2ILMUYBWPE5OOW2ARK3YTA6M6YQDMHPM55ECH5YPLXAUA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: IKYKESLO7UCB4BRMMGYBSGNE6MF73OAD6LUIAAXUR5VLLUCGLXWA) context deadline exceeded ======LOGS====== Dec 25 09:44:07.264250362 (474h15m7.135333407s) [upload=416292]: Starting upload Dec 25 09:44:07.264302622 (474h15m7.135385657s) [upload=416292]: Done starting upload Dec 25 09:44:07.264491972 (474h15m7.135575007s) [upload=416292]: Uploading segments... Dec 25 09:44:07.264540494 (474h15m7.135623529s) [upload=416292]: Got next segment. Inline: false Dec 25 09:44:07.264563668 (474h15m7.135646703s) [upload=416292]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:44:07.264568638 (474h15m7.135651673s) [upload=416292]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:44:07.508329921 (474h15m7.379412966s) [seg_pos={0 0}, upload=416292]: Begin upload segment... Dec 25 09:44:07.508371269 (474h15m7.379454304s) [seg_pos={0 0}, upload=416292]: Done begin upload segment. Dec 25 09:44:07.508395897 (474h15m7.379478932s) [noise=true , address=145.79.227.11:4729 , piece_id=E527Q3N43SKXLCIWASMKTQZ5WVNYKITNHNZUUXWARVSMHZPOAVSA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.508532487 (474h15m7.379615522s) [noise=true , address=145.79.227.17:4971 , piece_id=AK7OC6NMAZAXP2CA2266AM4BDWDADJ73VLPTDIXYWQ2K4N32GT2A, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.509394097 (474h15m7.380477142s) [noise=true , address=145.79.227.45:4230 , piece_id=QI3QODZPMFQGWZ7CFGJYPDVLS5MSTDL6O3VTIUTF4KEBN2DTQTNA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.509856548 (474h15m7.380939583s) [noise=true , address=145.79.227.29:4942 , piece_id=MVUTYJNMHSNZOMBCAWVV65AIAQD7C3NWKRPJ2EOPLLXLMMR5EKXQ, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.510410062 (474h15m7.381493097s) [noise=true , address=145.79.227.52:4447 , piece_id=5YBWBH72YFTTTICIEWFCREFN5GLSMYOKWGCM3LBLMMA7ZUCRKN2A, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.510884334 (474h15m7.38196738s) [noise=true , address=145.79.227.41:4087 , piece_id=KOQTJX5FLYBAVT5XZSQTZ6EAUDXNWC2A3WZPZTZFFCAA3TC7H6QA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.511552066 (474h15m7.382635111s) [noise=true , address=145.79.227.38:4979 , piece_id=C3UTDMVX22JDERB36UGH46TMI2WBKEBP56SUT5Q7KBO4URNRZSMA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.511760623 (474h15m7.382843658s) [noise=true , address=45.91.116.11:4120 , piece_id=HFA4RRO2ILMUYBWPE5OOW2ARK3YTA6M6YQDMHPM55ECH5YPLXAUA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.512685765 (474h15m7.38376881s) [noise=true , address=145.79.227.46:4601 , piece_id=VIKTFGPXOP35APA2UCB7H3MVLIT6BDZ52MM52MVYU3PHIEAZLRPA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.512793029 (474h15m7.383876074s) [noise=true , address=145.79.227.39:4015 , piece_id=C5OB5NN662K6JHYXZDYCQHQWVVD64YLGYZWXL4WZG7JWAGCHTASQ, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.513667013 (474h15m7.384750048s) [noise=true , address=145.79.227.40:4049 , piece_id=T5ZOIDVERBPRVCO2U2YOUSKOTGSKCNXIEDR7QC2NZJ7GRFVJKE6A, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.514128802 (474h15m7.385211837s) [noise=true , address=145.79.227.2:4113 , piece_id=BG7ABTYXGZDPWRSQ56RJLIUWIYNHJE7HADVTKYBMUGNUPWJP3ZGA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.514196191 (474h15m7.385279236s) [noise=true , address=77.90.31.14:4661 , piece_id=MKFZ7SAXQO7EX2GYZM5XIPVVIPZNVCTOVF4MDAV7U7SPN5NSENTQ, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.514614277 (474h15m7.385697322s) [noise=true , address=77.90.31.11:4556 , piece_id=IKYKESLO7UCB4BRMMGYBSGNE6MF73OAD6LUIAAXUR5VLLUCGLXWA, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.514936030 (474h15m7.386019065s) [noise=true , address=45.91.116.8:4047 , piece_id=3UNU3PA25FU4IJR2JK3TFMU4QW3FE2DXJ6QB4PF7OAXLAFJ4LG6Q, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:07.515285866 (474h15m7.386368901s) [noise=true , address=45.91.116.6:4974 , piece_id=LFVW3SICWO5U22K3WCSKWTOMLP2JKZBL3E4XOZRONKBQYRRON5MQ, seg_pos={0 0}, upload=416292]: Uploading piece... Dec 25 09:44:09.134358899 (474h15m9.005441944s) [upload=416292]: Next returned nil segment Dec 25 09:44:09.134379950 (474h15m9.005463025s) [upload=416292]: Waiting for error group managing segments... Dec 25 09:44:09.516963432 (474h15m9.388046477s) [noise=true , address=145.79.227.11:4729 , piece_id=E527Q3N43SKXLCIWASMKTQZ5WVNYKITNHNZUUXWARVSMHZPOAVSA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.525132447 (474h15m9.396215492s) [noise=true , address=145.79.227.17:4971 , piece_id=AK7OC6NMAZAXP2CA2266AM4BDWDADJ73VLPTDIXYWQ2K4N32GT2A, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.525291330 (474h15m9.396374365s) [noise=true , address=145.79.227.38:4979 , piece_id=C3UTDMVX22JDERB36UGH46TMI2WBKEBP56SUT5Q7KBO4URNRZSMA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.525644662 (474h15m9.396727697s) [noise=true , address=145.79.227.29:4942 , piece_id=MVUTYJNMHSNZOMBCAWVV65AIAQD7C3NWKRPJ2EOPLLXLMMR5EKXQ, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.526018924 (474h15m9.397101969s) [noise=true , address=145.79.227.46:4601 , piece_id=VIKTFGPXOP35APA2UCB7H3MVLIT6BDZ52MM52MVYU3PHIEAZLRPA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.526269031 (474h15m9.397352066s) [noise=true , address=145.79.227.39:4015 , piece_id=C5OB5NN662K6JHYXZDYCQHQWVVD64YLGYZWXL4WZG7JWAGCHTASQ, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.526944357 (474h15m9.398027392s) [noise=true , address=145.79.227.2:4113 , piece_id=BG7ABTYXGZDPWRSQ56RJLIUWIYNHJE7HADVTKYBMUGNUPWJP3ZGA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.530206759 (474h15m9.401289804s) [noise=true , address=145.79.227.45:4230 , piece_id=QI3QODZPMFQGWZ7CFGJYPDVLS5MSTDL6O3VTIUTF4KEBN2DTQTNA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.538734367 (474h15m9.409817412s) [noise=true , address=145.79.227.41:4087 , piece_id=KOQTJX5FLYBAVT5XZSQTZ6EAUDXNWC2A3WZPZTZFFCAA3TC7H6QA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.544202299 (474h15m9.415285344s) [noise=true , address=145.79.227.52:4447 , piece_id=5YBWBH72YFTTTICIEWFCREFN5GLSMYOKWGCM3LBLMMA7ZUCRKN2A, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.731088561 (474h15m9.602171606s) [noise=true , address=77.90.31.14:4661 , piece_id=MKFZ7SAXQO7EX2GYZM5XIPVVIPZNVCTOVF4MDAV7U7SPN5NSENTQ, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:09.772766583 (474h15m9.643849628s) [noise=true , address=45.91.116.11:4120 , piece_id=HFA4RRO2ILMUYBWPE5OOW2ARK3YTA6M6YQDMHPM55ECH5YPLXAUA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:10.165733478 (474h15m10.036816523s) [noise=true , address=77.90.31.11:4556 , piece_id=IKYKESLO7UCB4BRMMGYBSGNE6MF73OAD6LUIAAXUR5VLLUCGLXWA, seg_pos={0 0}, upload=416292]: Done uploading piece. err: <nil> Dec 25 09:44:10.165760330 (474h15m10.036843365s) [seg_pos={0 0}, upload=416292]: Segment reached optimal threshold of 13 pieces. Dec 25 09:44:10.166041665 (474h15m10.0371247s) [noise=true , address=45.91.116.6:4974 , piece_id=LFVW3SICWO5U22K3WCSKWTOMLP2JKZBL3E4XOZRONKBQYRRON5MQ, seg_pos={0 0}, upload=416292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 25 09:44:10.166051814 (474h15m10.037134859s) [noise=true , address=45.91.116.6:4974 , piece_id=LFVW3SICWO5U22K3WCSKWTOMLP2JKZBL3E4XOZRONKBQYRRON5MQ, seg_pos={0 0}, upload=416292]: Long tail cancellation due to optimal threshold Dec 25 09:44:10.166225455 (474h15m10.03730851s) [noise=true , address=45.91.116.8:4047 , piece_id=3UNU3PA25FU4IJR2JK3TFMU4QW3FE2DXJ6QB4PF7OAXLAFJ4LG6Q, seg_pos={0 0}, upload=416292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:34726->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:34726->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34726->45.91.116.8:4047: use of closed network connection Dec 25 09:44:10.166237518 (474h15m10.037320563s) [noise=true , address=45.91.116.8:4047 , piece_id=3UNU3PA25FU4IJR2JK3TFMU4QW3FE2DXJ6QB4PF7OAXLAFJ4LG6Q, seg_pos={0 0}, upload=416292]: Long tail cancellation due to optimal threshold Dec 25 09:44:10.166348600 (474h15m10.037431645s) [noise=true , address=145.79.227.40:4049 , piece_id=T5ZOIDVERBPRVCO2U2YOUSKOTGSKCNXIEDR7QC2NZJ7GRFVJKE6A, seg_pos={0 0}, upload=416292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 09:44:10.166358769 (474h15m10.037441814s) [noise=true , address=145.79.227.40:4049 , piece_id=T5ZOIDVERBPRVCO2U2YOUSKOTGSKCNXIEDR7QC2NZJ7GRFVJKE6A, seg_pos={0 0}, upload=416292]: Long tail cancellation due to optimal threshold Dec 25 09:44:10.166380861 (474h15m10.037463907s) [seg_pos={0 0}, upload=416292]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:44:10.166466955 (474h15m10.03755s) [upload=416292]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:44:10.166474319 (474h15m10.037557364s) [upload=416292]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:44:10.450581867 (474h15m10.321664922s) [upload=416292]: Done uploading segments... Dec 25 09:44:10.450603749 (474h15m10.321686784s) [upload=416292]: Upload finished. err: <nil> Duration: 30.003414651s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 4.072158819s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:43:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.072184579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.........X...........X...............] duration: 5.731392313s trace: https://jaeger.dev.storj.io/trace/795e298d20174ea1 FAIL (2025-12-25 09:43:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ]: Done starting upload Dec 25 09:33:01.262724874 (474h4m1.133807919s) [upload=416129]: Uploading segments... Dec 25 09:33:01.262992493 (474h4m1.134075538s) [upload=416129]: Got next segment. Inline: false Dec 25 09:33:01.263043731 (474h4m1.134126776s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:33:01.263050604 (474h4m1.134133649s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:33:01.511625191 (474h4m1.382708226s) [seg_pos={0 0}, upload=416129]: Begin upload segment... Dec 25 09:33:01.511677211 (474h4m1.382760246s) [seg_pos={0 0}, upload=416129]: Done begin upload segment. Dec 25 09:33:01.511706086 (474h4m1.382789131s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.520762050 (474h4m1.391845095s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.522962850 (474h4m1.394045895s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.528425993 (474h4m1.399509038s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.531387723 (474h4m1.402470768s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.532761669 (474h4m1.403844715s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535007145 (474h4m1.40609019s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.535436131 (474h4m1.406519176s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.536867767 (474h4m1.407950812s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.537459183 (474h4m1.408542228s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.538526586 (474h4m1.409609631s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.540585998 (474h4m1.411669043s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542237873 (474h4m1.413320948s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.542542713 (474h4m1.413625748s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.545729362 (474h4m1.416812407s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:01.546027639 (474h4m1.417110684s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Uploading piece... Dec 25 09:33:03.159768330 (474h4m3.030851375s) [upload=416129]: Next returned nil segment Dec 25 09:33:03.159785773 (474h4m3.030868808s) [upload=416129]: Waiting for error group managing segments... Dec 25 09:33:03.547727689 (474h4m3.418810734s) [noise=true , address=145.79.227.15:4876 , piece_id=C5CJCEWKFJD7MEZ2SZHO7DBNVGZCYDKZJFZELZRR55R225GMC2TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.548444835 (474h4m3.41952789s) [noise=true , address=145.79.227.20:4437 , piece_id=3F5MHHTTAYBWZQXKCDZWI2CAFL6POZKPDMCLUU6OJVHT2KXVZJAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.551962984 (474h4m3.423046039s) [noise=true , address=145.79.227.41:4086 , piece_id=YHWKTB2L5RRPDSA5CBGRZLYKC6X7RZQJJQRXO7A6L2QOPFNS5EAQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.560368159 (474h4m3.431451204s) [noise=true , address=145.79.227.43:4159 , piece_id=63WOEN56QBF6XEPD2D4Y2Z24KMSDMKWFGR5MZAM5DUBZDENOR5TA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.562870935 (474h4m3.43395398s) [noise=true , address=145.79.227.37:4565 , piece_id=SHAWCQGEMDXIVEIAQX3UCPA2PPQ4BGUVZXQMKHLDAVVF4R6YF7UA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.563062640 (474h4m3.434145705s) [noise=true , address=145.79.227.34:4907 , piece_id=64HKKTS6XXBD5VQX2WXM2BNKOVN2ODRK4M2QEUQOJWK4WXMI6OMQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.573975500 (474h4m3.445058545s) [noise=true , address=145.79.227.4:4186 , piece_id=M5YH3QWBF4M3DSBAK3GPLSI2EYXIM7NILWO6R2QMPF7W5VO3DO6A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.582677770 (474h4m3.453760815s) [noise=true , address=145.79.227.17:4970 , piece_id=HHZHY7KKCO3KEVWREHYTICOJ6M5DLH5TR2NL7VFTMIQSONQ2IQIQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.588464799 (474h4m3.459547844s) [noise=true , address=145.79.227.47:4265 , piece_id=HYBC7LNY3ZVHW3NBXYUINNVZBQWZZARCJIX64ZCQC4NHSCTSP43A, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.712544699 (474h4m3.583627744s) [noise=true , address=77.90.31.13:4629 , piece_id=TXIOUXOCF3NAC3XDTINCQDO6VUYGRCHB7RRZ2VULV4A42FU2RXPA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.851482769 (474h4m3.722565814s) [noise=true , address=45.91.116.6:4973 , piece_id=J5WBXHRZ634A57H3WMB6WYM2RNPDVSSKNBWD3WPT5N2OHSIETXZA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.882255556 (474h4m3.753338601s) [noise=true , address=45.91.116.14:4189 , piece_id=MU63DJXIVNPFPZYC6QTIYJ6PQZXG3H4XPAM7UEFCGGHZ3VKYFPBA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918839348 (474h4m3.789922403s) [noise=true , address=45.91.116.9:4261 , piece_id=VMJBON2GCIN2LP4C5V4YYTBZX5MMMCOZYFDV7QVUHNLY6A5JWYDQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: <nil> Dec 25 09:33:03.918864156 (474h4m3.789947201s) [seg_pos={0 0}, upload=416129]: Segment reached optimal threshold of 13 pieces. Dec 25 09:33:03.919269627 (474h4m3.790352672s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 25 09:33:03.919280899 (474h4m3.790363944s) [noise=true , address=45.91.116.10:4082 , piece_id=B6JOLZGFK4FJOIWTCIBC3KCHW36BNDSLI6K23I36ETV4GSRQD2XQ, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919532328 (474h4m3.790615373s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38690->77.90.31.10:4518: use of closed network connection Dec 25 09:33:03.919541034 (474h4m3.790624079s) [noise=true , address=77.90.31.10:4518 , piece_id=CXATQATVQLDL4D5VRSIBLOQLFYTZE22IFFCKUEK4L532HEVVQMJA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919588836 (474h4m3.790671871s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35168->45.91.116.8:4045: use of closed network connection Dec 25 09:33:03.919609945 (474h4m3.79069299s) [noise=true , address=45.91.116.8:4045 , piece_id=G3XTKRRCSTEDWBILTD73M3ASGY2YE24QFTLZZWAUXRCIGGEUBGFA, seg_pos={0 0}, upload=416129]: Long tail cancellation due to optimal threshold Dec 25 09:33:03.919639131 (474h4m3.790722176s) [seg_pos={0 0}, upload=416129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:33:03.919766364 (474h4m3.790849409s) [upload=416129]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:33:03.919775921 (474h4m3.790858977s) [upload=416129]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:33:04.215163028 (474h4m4.086246073s) [upload=416129]: Done uploading segments... Dec 25 09:33:04.215194258 (474h4m4.086277303s) [upload=416129]: Upload finished. err: <nil> Duration: 5.731426689s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)