Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X..X.XXXXXX] duration: 5.001765188s trace: https://jaeger.dev.storj.io/trace/65afc4b5fee00c53 FAIL (2025-12-15 13:12:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001781631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............................X.....X..X..X..........X..XX..X] duration: 11.892702367s trace: https://jaeger.dev.storj.io/trace/764546275c7f76e2 FAIL (2025-12-15 13:12:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.892731462s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X..X.XXXXX] duration: 5.00138774s trace: https://jaeger.dev.storj.io/trace/5fb4f718acaa8383 FAIL (2025-12-15 13:12:23 UTC) orbiter warning: context deadline exceeded Duration: 5.001412136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 20.037164709s trace: https://jaeger.dev.storj.io/trace/44aa0b9bc841ed28 FAIL (2025-12-15 13:11:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 13:10:22.936146585 (237h41m22.807229621s) [upload=208411]: Starting upload Dec 15 13:10:22.936205197 (237h41m22.807288232s) [upload=208411]: Done starting upload Dec 15 13:10:22.936379990 (237h41m22.807463035s) [upload=208411]: Uploading segments... Dec 15 13:10:22.936494678 (237h41m22.807577723s) [upload=208411]: Got next segment. Inline: false Dec 15 13:10:22.936535126 (237h41m22.807618171s) [upload=208411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:10:22.936541508 (237h41m22.807624553s) [upload=208411]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:10:23.296518274 (237h41m23.167601319s) [seg_pos={0 0}, upload=208411]: Begin upload segment... Dec 15 13:10:23.296619507 (237h41m23.167702562s) [seg_pos={0 0}, upload=208411]: Done begin upload segment. Dec 15 13:10:23.296630548 (237h41m23.167713593s) [noise=true , address=145.79.227.23:4546 , piece_id=62ABDSYGDTUAG2JNFKQ2SJNJQBEZOLLS7QET4GBVHRKHYJTVWEIQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.296663020 (237h41m23.167746065s) [noise=true , address=145.79.227.43:4157 , piece_id=LD4GQAPWJCLN6OV7RZOHSD3GPNL2V57X5FRBO5MVUPKUCTR6MA2Q, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.297335650 (237h41m23.168418695s) [noise=true , address=145.79.227.24:4582 , piece_id=U2I7M2L4HC3CIS5ZFEFV3MCHRNMOM6EJS4B3VZZG3DDMXDCBQGXQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.297777281 (237h41m23.168860326s) [noise=true , address=145.79.227.49:4338 , piece_id=ZML6Z27KG7KYGQ5AX2DXF62FH5TGESX7U3KLAV6PL2D6D5Z2EX2Q, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.298366924 (237h41m23.169449969s) [noise=true , address=145.79.227.28:4725 , piece_id=SDVKLBWWY5U34XXLUOAT5MUGMTFA7ASGPJLIRPUY7RRKCORBMP6A, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.298867637 (237h41m23.169950682s) [noise=true , address=145.79.227.29:4942 , piece_id=K5IBN2KZTKP4YV4T2B4KDITA2KJAA3EYUFWO77ZBZ4ORHNLOO5EA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.299400573 (237h41m23.170483618s) [noise=true , address=77.90.31.19:4841 , piece_id=NCW6X4WQHV3BPESGCMYE2Y433LSQP3PEAKJP7TL66X7OYDXORZSQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.299948545 (237h41m23.17103158s) [noise=true , address=77.90.31.15:4697 , piece_id=FLKBEQRCPTLFPNNX6D6LJ5QS4DXIO7Z5DQCM3PLJIXZDSEHLENYQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.301856959 (237h41m23.172940005s) [noise=true , address=77.90.31.21:4877 , piece_id=PRIGQNMHZ2NXWAJ4AKXSV74XUSS4TCL5UCLC7CDILQIC2DBTKYKQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.303635377 (237h41m23.174718422s) [noise=true , address=145.79.227.52:4447 , piece_id=CG2IMM7UFXM6NZZR4NDRZTJVI275PSKTX5TCJKL4EP3BBP36OUZA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.305538049 (237h41m23.176621094s) [noise=true , address=145.79.227.35:4482 , piece_id=P5EJEWJ5EOMB6TOIY5BOU434NVIKR2ZUHU5JSB52WD7PDOOZ6QUQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.307292781 (237h41m23.178375826s) [noise=true , address=145.79.227.27:4690 , piece_id=VOG4GWMHGVJ2ES3NN2LPU7CWFWRILSTYD3XQ5GDX23NTJVWFPCBA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.308830289 (237h41m23.179913334s) [noise=true , address=145.79.227.40:4051 , piece_id=THFNWKBW7ZKVNW725QDGP2P4BVSPLE55Q7ZI3I4ACR3AHH4FKPZA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.309960540 (237h41m23.181043585s) [noise=true , address=145.79.227.42:4121 , piece_id=WF4ILYACBSWE3DFQAT5JCN726L6UHTOFRY7M34JOB5ZPM6U4AEFA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.311307836 (237h41m23.182390881s) [noise=true , address=145.79.227.50:4374 , piece_id=WWATAHGVSZVLZCASF53EA6PC5JXKMFZOX4VD2EGUP3OUL6RUXN5A, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.312165850 (237h41m23.183248885s) [noise=true , address=145.79.227.32:4836 , piece_id=YGHIKJR5ZAR3E4644QRQX7RZQNNUYMRC4WLNUHMS5JSFHMDIDKOQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:24.411408622 (237h41m24.282491668s) [noise=true , address=77.90.31.19:4841 , piece_id=NCW6X4WQHV3BPESGCMYE2Y433LSQP3PEAKJP7TL66X7OYDXORZSQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 13:10:24.414338882 (237h41m24.285421957s) [noise=true , address=77.90.31.21:4877 , piece_id=PRIGQNMHZ2NXWAJ4AKXSV74XUSS4TCL5UCLC7CDILQIC2DBTKYKQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 13:10:25.878779314 (237h41m25.749862359s) [upload=208411]: Next returned nil segment Dec 15 13:10:25.878793591 (237h41m25.749876636s) [upload=208411]: Waiting for error group managing segments... Dec 15 13:10:26.312910053 (237h41m26.183993088s) [noise=true , address=145.79.227.29:4942 , piece_id=K5IBN2KZTKP4YV4T2B4KDITA2KJAA3EYUFWO77ZBZ4ORHNLOO5EA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.312963735 (237h41m26.18404678s) [noise=true , address=145.79.227.24:4582 , piece_id=U2I7M2L4HC3CIS5ZFEFV3MCHRNMOM6EJS4B3VZZG3DDMXDCBQGXQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.314112412 (237h41m26.185195457s) [noise=true , address=145.79.227.40:4051 , piece_id=THFNWKBW7ZKVNW725QDGP2P4BVSPLE55Q7ZI3I4ACR3AHH4FKPZA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.318504126 (237h41m26.189587181s) [noise=true , address=145.79.227.27:4690 , piece_id=VOG4GWMHGVJ2ES3NN2LPU7CWFWRILSTYD3XQ5GDX23NTJVWFPCBA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.324069963 (237h41m26.195153008s) [noise=true , address=145.79.227.42:4121 , piece_id=WF4ILYACBSWE3DFQAT5JCN726L6UHTOFRY7M34JOB5ZPM6U4AEFA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.328826321 (237h41m26.199909366s) [noise=true , address=145.79.227.28:4725 , piece_id=SDVKLBWWY5U34XXLUOAT5MUGMTFA7ASGPJLIRPUY7RRKCORBMP6A, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.330581613 (237h41m26.201664659s) [noise=true , address=145.79.227.43:4157 , piece_id=LD4GQAPWJCLN6OV7RZOHSD3GPNL2V57X5FRBO5MVUPKUCTR6MA2Q, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.330702824 (237h41m26.201785869s) [noise=true , address=145.79.227.49:4338 , piece_id=ZML6Z27KG7KYGQ5AX2DXF62FH5TGESX7U3KLAV6PL2D6D5Z2EX2Q, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.337693275 (237h41m26.20877632s) [noise=true , address=145.79.227.50:4374 , piece_id=WWATAHGVSZVLZCASF53EA6PC5JXKMFZOX4VD2EGUP3OUL6RUXN5A, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.339113059 (237h41m26.210196104s) [noise=true , address=145.79.227.35:4482 , piece_id=P5EJEWJ5EOMB6TOIY5BOU434NVIKR2ZUHU5JSB52WD7PDOOZ6QUQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:27.099370167 (237h41m26.970453212s) [noise=true , address=77.90.31.15:4697 , piece_id=FLKBEQRCPTLFPNNX6D6LJ5QS4DXIO7Z5DQCM3PLJIXZDSEHLENYQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:27.191718536 (237h41m27.062801581s) [noise=true , address=145.79.227.23:4546 , piece_id=62ABDSYGDTUAG2JNFKQ2SJNJQBEZOLLS7QET4GBVHRKHYJTVWEIQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:27.212934335 (237h41m27.08401738s) [noise=true , address=145.79.227.52:4447 , piece_id=CG2IMM7UFXM6NZZR4NDRZTJVI275PSKTX5TCJKL4EP3BBP36OUZA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:27.212961898 (237h41m27.084044943s) [seg_pos={0 0}, upload=208411]: Segment reached optimal threshold of 13 pieces. Dec 15 13:10:27.213361989 (237h41m27.084445034s) [noise=true , address=145.79.227.32:4836 , piece_id=YGHIKJR5ZAR3E4644QRQX7RZQNNUYMRC4WLNUHMS5JSFHMDIDKOQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 13:10:27.213373752 (237h41m27.084456797s) [noise=true , address=145.79.227.32:4836 , piece_id=YGHIKJR5ZAR3E4644QRQX7RZQNNUYMRC4WLNUHMS5JSFHMDIDKOQ, seg_pos={0 0}, upload=208411]: Long tail cancellation due to optimal threshold Dec 15 13:10:27.213405813 (237h41m27.084488859s) [seg_pos={0 0}, upload=208411]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 13:10:27.213504120 (237h41m27.084587165s) [upload=208411]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 13:10:27.213511164 (237h41m27.084594209s) [upload=208411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 13:10:27.638554040 (237h41m27.509637085s) [upload=208411]: Done uploading segments... Dec 15 13:10:27.638579649 (237h41m27.509662694s) [upload=208411]: Upload finished. err: <nil> Duration: 20.037191671s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [.XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X..X.XXXX] duration: 5.001426033s trace: https://jaeger.dev.storj.io/trace/6284c42618d26113 FAIL (2025-12-15 13:11:54 UTC) orbiter warning: context deadline exceeded Duration: 5.001449416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................X.....X..X..X..........X..XX..] duration: 21.467922642s trace: https://jaeger.dev.storj.io/trace/4f1d53e3a62dd518 FAIL (2025-12-15 13:11:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.467941658s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................X.....X..X..X..........X..XX.] duration: 23.937361493s trace: https://jaeger.dev.storj.io/trace/4f1d53e3a62dd518 FAIL (2025-12-15 13:11:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.937395287s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [..XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X..X.XXX] duration: 5.001878967s trace: https://jaeger.dev.storj.io/trace/5d10a208d49848ee FAIL (2025-12-15 13:11:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001905488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................................X.....X..X..X..........X..XX] duration: 30.002183555s trace: https://jaeger.dev.storj.io/trace/4f1d53e3a62dd518 FAIL (2025-12-15 13:11:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002215105s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X..X.XX] duration: 5.001627917s trace: https://jaeger.dev.storj.io/trace/409ad475a7f2bc8d FAIL (2025-12-15 13:10:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00165559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003346254s trace: https://jaeger.dev.storj.io/trace/44aa0b9bc841ed28 FAIL (2025-12-15 13:10:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 15 13:10:22.936146585 (237h41m22.807229621s) [upload=208411]: Starting upload Dec 15 13:10:22.936205197 (237h41m22.807288232s) [upload=208411]: Done starting upload Dec 15 13:10:22.936379990 (237h41m22.807463035s) [upload=208411]: Uploading segments... Dec 15 13:10:22.936494678 (237h41m22.807577723s) [upload=208411]: Got next segment. Inline: false Dec 15 13:10:22.936535126 (237h41m22.807618171s) [upload=208411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 13:10:22.936541508 (237h41m22.807624553s) [upload=208411]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 13:10:23.296518274 (237h41m23.167601319s) [seg_pos={0 0}, upload=208411]: Begin upload segment... Dec 15 13:10:23.296619507 (237h41m23.167702562s) [seg_pos={0 0}, upload=208411]: Done begin upload segment. Dec 15 13:10:23.296630548 (237h41m23.167713593s) [noise=true , address=145.79.227.23:4546 , piece_id=62ABDSYGDTUAG2JNFKQ2SJNJQBEZOLLS7QET4GBVHRKHYJTVWEIQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.296663020 (237h41m23.167746065s) [noise=true , address=145.79.227.43:4157 , piece_id=LD4GQAPWJCLN6OV7RZOHSD3GPNL2V57X5FRBO5MVUPKUCTR6MA2Q, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.297335650 (237h41m23.168418695s) [noise=true , address=145.79.227.24:4582 , piece_id=U2I7M2L4HC3CIS5ZFEFV3MCHRNMOM6EJS4B3VZZG3DDMXDCBQGXQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.297777281 (237h41m23.168860326s) [noise=true , address=145.79.227.49:4338 , piece_id=ZML6Z27KG7KYGQ5AX2DXF62FH5TGESX7U3KLAV6PL2D6D5Z2EX2Q, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.298366924 (237h41m23.169449969s) [noise=true , address=145.79.227.28:4725 , piece_id=SDVKLBWWY5U34XXLUOAT5MUGMTFA7ASGPJLIRPUY7RRKCORBMP6A, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.298867637 (237h41m23.169950682s) [noise=true , address=145.79.227.29:4942 , piece_id=K5IBN2KZTKP4YV4T2B4KDITA2KJAA3EYUFWO77ZBZ4ORHNLOO5EA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.299400573 (237h41m23.170483618s) [noise=true , address=77.90.31.19:4841 , piece_id=NCW6X4WQHV3BPESGCMYE2Y433LSQP3PEAKJP7TL66X7OYDXORZSQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.299948545 (237h41m23.17103158s) [noise=true , address=77.90.31.15:4697 , piece_id=FLKBEQRCPTLFPNNX6D6LJ5QS4DXIO7Z5DQCM3PLJIXZDSEHLENYQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.301856959 (237h41m23.172940005s) [noise=true , address=77.90.31.21:4877 , piece_id=PRIGQNMHZ2NXWAJ4AKXSV74XUSS4TCL5UCLC7CDILQIC2DBTKYKQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.303635377 (237h41m23.174718422s) [noise=true , address=145.79.227.52:4447 , piece_id=CG2IMM7UFXM6NZZR4NDRZTJVI275PSKTX5TCJKL4EP3BBP36OUZA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.305538049 (237h41m23.176621094s) [noise=true , address=145.79.227.35:4482 , piece_id=P5EJEWJ5EOMB6TOIY5BOU434NVIKR2ZUHU5JSB52WD7PDOOZ6QUQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.307292781 (237h41m23.178375826s) [noise=true , address=145.79.227.27:4690 , piece_id=VOG4GWMHGVJ2ES3NN2LPU7CWFWRILSTYD3XQ5GDX23NTJVWFPCBA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.308830289 (237h41m23.179913334s) [noise=true , address=145.79.227.40:4051 , piece_id=THFNWKBW7ZKVNW725QDGP2P4BVSPLE55Q7ZI3I4ACR3AHH4FKPZA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.309960540 (237h41m23.181043585s) [noise=true , address=145.79.227.42:4121 , piece_id=WF4ILYACBSWE3DFQAT5JCN726L6UHTOFRY7M34JOB5ZPM6U4AEFA, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.311307836 (237h41m23.182390881s) [noise=true , address=145.79.227.50:4374 , piece_id=WWATAHGVSZVLZCASF53EA6PC5JXKMFZOX4VD2EGUP3OUL6RUXN5A, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:23.312165850 (237h41m23.183248885s) [noise=true , address=145.79.227.32:4836 , piece_id=YGHIKJR5ZAR3E4644QRQX7RZQNNUYMRC4WLNUHMS5JSFHMDIDKOQ, seg_pos={0 0}, upload=208411]: Uploading piece... Dec 15 13:10:24.411408622 (237h41m24.282491668s) [noise=true , address=77.90.31.19:4841 , piece_id=NCW6X4WQHV3BPESGCMYE2Y433LSQP3PEAKJP7TL66X7OYDXORZSQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 15 13:10:24.414338882 (237h41m24.285421957s) [noise=true , address=77.90.31.21:4877 , piece_id=PRIGQNMHZ2NXWAJ4AKXSV74XUSS4TCL5UCLC7CDILQIC2DBTKYKQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 15 13:10:25.878779314 (237h41m25.749862359s) [upload=208411]: Next returned nil segment Dec 15 13:10:25.878793591 (237h41m25.749876636s) [upload=208411]: Waiting for error group managing segments... Dec 15 13:10:26.312910053 (237h41m26.183993088s) [noise=true , address=145.79.227.29:4942 , piece_id=K5IBN2KZTKP4YV4T2B4KDITA2KJAA3EYUFWO77ZBZ4ORHNLOO5EA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.312963735 (237h41m26.18404678s) [noise=true , address=145.79.227.24:4582 , piece_id=U2I7M2L4HC3CIS5ZFEFV3MCHRNMOM6EJS4B3VZZG3DDMXDCBQGXQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.314112412 (237h41m26.185195457s) [noise=true , address=145.79.227.40:4051 , piece_id=THFNWKBW7ZKVNW725QDGP2P4BVSPLE55Q7ZI3I4ACR3AHH4FKPZA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.318504126 (237h41m26.189587181s) [noise=true , address=145.79.227.27:4690 , piece_id=VOG4GWMHGVJ2ES3NN2LPU7CWFWRILSTYD3XQ5GDX23NTJVWFPCBA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.324069963 (237h41m26.195153008s) [noise=true , address=145.79.227.42:4121 , piece_id=WF4ILYACBSWE3DFQAT5JCN726L6UHTOFRY7M34JOB5ZPM6U4AEFA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.328826321 (237h41m26.199909366s) [noise=true , address=145.79.227.28:4725 , piece_id=SDVKLBWWY5U34XXLUOAT5MUGMTFA7ASGPJLIRPUY7RRKCORBMP6A, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.330581613 (237h41m26.201664659s) [noise=true , address=145.79.227.43:4157 , piece_id=LD4GQAPWJCLN6OV7RZOHSD3GPNL2V57X5FRBO5MVUPKUCTR6MA2Q, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.330702824 (237h41m26.201785869s) [noise=true , address=145.79.227.49:4338 , piece_id=ZML6Z27KG7KYGQ5AX2DXF62FH5TGESX7U3KLAV6PL2D6D5Z2EX2Q, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.337693275 (237h41m26.20877632s) [noise=true , address=145.79.227.50:4374 , piece_id=WWATAHGVSZVLZCASF53EA6PC5JXKMFZOX4VD2EGUP3OUL6RUXN5A, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:26.339113059 (237h41m26.210196104s) [noise=true , address=145.79.227.35:4482 , piece_id=P5EJEWJ5EOMB6TOIY5BOU434NVIKR2ZUHU5JSB52WD7PDOOZ6QUQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:27.099370167 (237h41m26.970453212s) [noise=true , address=77.90.31.15:4697 , piece_id=FLKBEQRCPTLFPNNX6D6LJ5QS4DXIO7Z5DQCM3PLJIXZDSEHLENYQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:27.191718536 (237h41m27.062801581s) [noise=true , address=145.79.227.23:4546 , piece_id=62ABDSYGDTUAG2JNFKQ2SJNJQBEZOLLS7QET4GBVHRKHYJTVWEIQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:27.212934335 (237h41m27.08401738s) [noise=true , address=145.79.227.52:4447 , piece_id=CG2IMM7UFXM6NZZR4NDRZTJVI275PSKTX5TCJKL4EP3BBP36OUZA, seg_pos={0 0}, upload=208411]: Done uploading piece. err: <nil> Dec 15 13:10:27.212961898 (237h41m27.084044943s) [seg_pos={0 0}, upload=208411]: Segment reached optimal threshold of 13 pieces. Dec 15 13:10:27.213361989 (237h41m27.084445034s) [noise=true , address=145.79.227.32:4836 , piece_id=YGHIKJR5ZAR3E4644QRQX7RZQNNUYMRC4WLNUHMS5JSFHMDIDKOQ, seg_pos={0 0}, upload=208411]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF Dec 15 13:10:27.213373752 (237h41m27.084456797s) [noise=true , address=145.79.227.32:4836 , piece_id=YGHIKJR5ZAR3E4644QRQX7RZQNNUYMRC4WLNUHMS5JSFHMDIDKOQ, seg_pos={0 0}, upload=208411]: Long tail cancellation due to optimal threshold Dec 15 13:10:27.213405813 (237h41m27.084488859s) [seg_pos={0 0}, upload=208411]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 13:10:27.213504120 (237h41m27.084587165s) [upload=208411]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 13:10:27.213511164 (237h41m27.084594209s) [upload=208411]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 13:10:27.638554040 (237h41m27.509637085s) [upload=208411]: Done uploading segments... Dec 15 13:10:27.638579649 (237h41m27.509662694s) [upload=208411]: Upload finished. err: <nil> Duration: 30.003442857s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [X...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X..X.X] duration: 5.001421343s trace: https://jaeger.dev.storj.io/trace/3fe313b3a67481d FAIL (2025-12-15 13:10:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00147265s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X..X.] duration: 4.056681116s trace: https://jaeger.dev.storj.io/trace/146a255cc1ab7500 FAIL (2025-12-15 13:10:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.056705202s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................X.....X..X..X..........X..X] duration: 30.001225055s trace: https://jaeger.dev.storj.io/trace/61713ef0ed6692b4 FAIL (2025-12-15 13:10:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001241907s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X..X] duration: 5.001439209s trace: https://jaeger.dev.storj.io/trace/146a255cc1ab7500 FAIL (2025-12-15 13:09:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001463353s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X.....X..X..X..........X..] duration: 24.528682081s trace: https://jaeger.dev.storj.io/trace/6d29e870a3cbdc2b FAIL (2025-12-15 13:09:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.528762104s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [..XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X..] duration: 5.000540368s trace: https://jaeger.dev.storj.io/trace/43ef98256ddea89b FAIL (2025-12-15 13:09:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000556198s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................X.....X..X..X..........X.] duration: 17.893863214s trace: https://jaeger.dev.storj.io/trace/6d29e870a3cbdc2b FAIL (2025-12-15 13:08:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.893883552s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [...XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X.] duration: 5.001155519s trace: https://jaeger.dev.storj.io/trace/43ef98256ddea89b FAIL (2025-12-15 13:08:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001175938s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................X.....X..X..X..........X] duration: 30.003328376s trace: https://jaeger.dev.storj.io/trace/6d29e870a3cbdc2b FAIL (2025-12-15 13:08:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.003357853s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [....XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.X] duration: 5.001863056s trace: https://jaeger.dev.storj.io/trace/43ef98256ddea89b FAIL (2025-12-15 13:07:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001901519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X.....X..X..X..........] duration: 18.355456856s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:07:18 UTC) orbiter warning: context deadline exceeded Duration: 18.355478486s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.....XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX.] duration: 4.935259091s trace: https://jaeger.dev.storj.io/trace/2d5a2bc383036a22 FAIL (2025-12-15 13:07:10 UTC) orbiter warning: context deadline exceeded Duration: 4.935300949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X.....X..X..X.........] duration: 24.320447675s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:06:59 UTC) orbiter warning: context deadline exceeded Duration: 24.320473213s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [......XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXXX] duration: 5.001102286s trace: https://jaeger.dev.storj.io/trace/2d5a2bc383036a22 FAIL (2025-12-15 13:06:29 UTC) orbiter warning: context deadline exceeded Duration: 5.001128908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X.....X..X..X........] duration: 29.893111112s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:06:25 UTC) orbiter warning: context deadline exceeded Duration: 29.893172891s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X.....X..X..X.......] duration: 16.398607105s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:05:38 UTC) orbiter warning: context deadline exceeded Duration: 16.398634346s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.......XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXXX] duration: 5.000797677s trace: https://jaeger.dev.storj.io/trace/35badeedf630e1bf FAIL (2025-12-15 13:05:19 UTC) orbiter warning: context deadline exceeded Duration: 5.000820551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X.....X..X..X......] duration: 18.573698396s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:05:04 UTC) orbiter warning: context deadline exceeded Duration: 18.573748561s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXXX] duration: 5.001338527s trace: https://jaeger.dev.storj.io/trace/75cfdc08dbb1716d FAIL (2025-12-15 13:04:55 UTC) orbiter warning: context deadline exceeded Duration: 5.001364496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X.....X..X..X.....] duration: 24.451619127s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:04:35 UTC) orbiter warning: context deadline exceeded Duration: 24.451646648s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXXX] duration: 5.001226664s trace: https://jaeger.dev.storj.io/trace/4345c35aa27fea71 FAIL (2025-12-15 13:03:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001254026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X.....X..X..X....] duration: 14.254912399s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:03:44 UTC) orbiter warning: context deadline exceeded Duration: 14.254943107s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXXX] duration: 5.002432048s trace: https://jaeger.dev.storj.io/trace/482a103337209da4 FAIL (2025-12-15 13:03:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002456866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X.....X..X..X...] duration: 18.081290736s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:02:55 UTC) orbiter warning: context deadline exceeded Duration: 18.081311014s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXXX] duration: 5.000571245s trace: https://jaeger.dev.storj.io/trace/2d96af529ac37530 FAIL (2025-12-15 13:02:36 UTC) orbiter warning: context deadline exceeded Duration: 5.000585754s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X.....X..X..X..] duration: 11.591363467s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:01:57 UTC) orbiter warning: context deadline exceeded Duration: 11.591390659s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X.....X..X..X.] duration: 13.285817251s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:01:39 UTC) orbiter warning: context deadline exceeded Duration: 13.285844613s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXXX] duration: 5.001354847s trace: https://jaeger.dev.storj.io/trace/625e311ff3004f3b FAIL (2025-12-15 13:01:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001375537s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................X.....X..X..X] duration: 30.001784677s trace: https://jaeger.dev.storj.io/trace/37b6dcfacbf8e1c9 FAIL (2025-12-15 13:01:21 UTC) orbiter warning: context deadline exceeded Duration: 30.001810826s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.....X..X..] duration: 15.08059939s trace: https://jaeger.dev.storj.io/trace/acc5e676b1715ed FAIL (2025-12-15 13:00:43 UTC) orbiter warning: context deadline exceeded Duration: 15.080620821s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXXX] duration: 5.001297279s trace: https://jaeger.dev.storj.io/trace/449e51e7a780f7e8 FAIL (2025-12-15 13:00:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001324129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X.....X..X.] duration: 27.369800542s trace: https://jaeger.dev.storj.io/trace/acc5e676b1715ed FAIL (2025-12-15 13:00:28 UTC) orbiter warning: context deadline exceeded Duration: 27.369818485s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [.X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXXX] duration: 5.001671631s trace: https://jaeger.dev.storj.io/trace/50d292750c895873 FAIL (2025-12-15 13:00:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001696678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 19.658595006s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:59:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 19.658622569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 17.635939041s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:59:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 17.635990119s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [..X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXXX] duration: 5.000748273s trace: https://jaeger.dev.storj.io/trace/57193b9b2791cda2 FAIL (2025-12-15 12:59:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000772209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X.....X..X] duration: 30.001850003s trace: https://jaeger.dev.storj.io/trace/acc5e676b1715ed FAIL (2025-12-15 12:59:22 UTC) orbiter warning: context deadline exceeded Duration: 30.001863739s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [...X.X..........XX...XX.XXXXXXXXXXXXXX.XXXXXXXXX.XX.XXXXXXXX] duration: 5.0015494s trace: https://jaeger.dev.storj.io/trace/7cefdd22db261583 FAIL (2025-12-15 12:58:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001576713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 10.930569662s trace: https://jaeger.dev.storj.io/trace/6e0710a87d2d6807 FAIL (2025-12-15 12:58:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA) context deadline exceeded ======LOGS====== Dec 15 12:20:01.862381976 (236h51m1.733465021s) [upload=207673]: Starting upload Dec 15 12:20:01.862499100 (236h51m1.733582145s) [upload=207673]: Done starting upload Dec 15 12:20:01.862793350 (236h51m1.733876415s) [upload=207673]: Uploading segments... Dec 15 12:20:01.862848976 (236h51m1.733932021s) [upload=207673]: Got next segment. Inline: false Dec 15 12:20:01.862895414 (236h51m1.733978459s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 15 12:20:01.862902578 (236h51m1.733985623s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 15 12:20:02.212326977 (236h51m2.083410022s) [seg_pos={0 0}, upload=207673]: Begin upload segment... Dec 15 12:20:02.212387622 (236h51m2.083470667s) [seg_pos={0 0}, upload=207673]: Done begin upload segment. Dec 15 12:20:02.212428911 (236h51m2.083511957s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.212624433 (236h51m2.083707478s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213448382 (236h51m2.084531468s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213970767 (236h51m2.085053812s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.213998580 (236h51m2.085081625s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214303280 (236h51m2.085386365s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214507499 (236h51m2.085590544s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214774658 (236h51m2.085857703s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.214938670 (236h51m2.086021725s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215158168 (236h51m2.086241213s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215424716 (236h51m2.086507761s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215562799 (236h51m2.086645834s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215889631 (236h51m2.086972676s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.215922614 (236h51m2.087005669s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216346841 (236h51m2.087429886s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:02.216801096 (236h51m2.087884141s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Uploading piece... Dec 15 12:20:04.122574133 (236h51m3.993657178s) [upload=207673]: Next returned nil segment Dec 15 12:20:04.122592848 (236h51m3.993675904s) [upload=207673]: Waiting for error group managing segments... Dec 15 12:20:04.555042394 (236h51m4.426125439s) [noise=true , address=77.90.31.11:4554 , piece_id=7PVYPC5ODQIIY77S7F3YWB3WS4H2WF7LQWZZMELFP3QDJH7IU6NA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.277053707 (236h51m5.148136792s) [noise=true , address=145.79.227.27:4691 , piece_id=C55UMMYNEI34DG5NPWTIWICD23SQ4ZL2OHRLP4YETVLMRBVJN5FQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.287841189 (236h51m5.158924234s) [noise=true , address=145.79.227.36:4518 , piece_id=HLKMQWC6A5BLG46GBT7IUXP7WEA3DPKNK4G4JOPVHSLW4RJORC5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367164034 (236h51m5.238247079s) [noise=true , address=145.79.227.45:4231 , piece_id=OWA4IFLGPSRJMAY7VH5OIQWW66JYTHBO2E6TCN3KPMN7I34653OA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.367447404 (236h51m5.238530449s) [noise=true , address=145.79.227.37:4566 , piece_id=REUFVVSV34GQBC372ONQ3POR7NWEULRKVUM4V64PQKY2KUVA7B7A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.374666820 (236h51m5.245749865s) [noise=true , address=145.79.227.38:4979 , piece_id=6ZRGLDYNN6MDBO4UDNQCE6VMXFA26VYPDDBIGLAR5L6AEBRBZIDA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:05.382667866 (236h51m5.253750911s) [noise=true , address=145.79.227.41:4086 , piece_id=AN2REZLKPV4QWJGCLGBCT5EXDJN7VYPE6576OZSOMBCY2Z7E3JJQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:07.930254305 (236h51m7.80133735s) [noise=true , address=145.79.227.46:4602 , piece_id=HLYO7WCUYYU76ENVUNEU7RO7FVTTZ2GRAQ7JS2WUBRHZ2WOEFZBA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.551545960 (236h51m8.422629015s) [noise=true , address=145.79.227.32:4836 , piece_id=ZIV6BLNSCRM6UG7TNMHZFRQ56GP3NC22QQEI57WCDHQZODT7CVIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.576551084 (236h51m8.447634129s) [noise=true , address=145.79.227.40:4051 , piece_id=4PSDVE2ETH2FTCRKKUSRQAK3E6MYFVFN3KYZWOSSOB3LLLWX7IIA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.700399725 (236h51m8.57148277s) [noise=true , address=45.91.116.8:4045 , piece_id=SCUMQNV5S243PHHSKGCK6HEWIKSLVVB67BR247KIMDWMUM4WSUGA, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:08.906210046 (236h51m8.777293092s) [noise=true , address=145.79.227.21:4474 , piece_id=SKUUOIPJFZLCSEY3TWKFSYHBPPCZAENM35GNQ6WGZLNVEXHBRK5A, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498845806 (236h51m9.369928851s) [noise=true , address=145.79.227.43:4158 , piece_id=3OER452MCSWXJN5OGO4MPG4QDM7PIF3YWYZWPFYOUNJ2Y5W4X6LQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: <nil> Dec 15 12:20:09.498871925 (236h51m9.36995497s) [seg_pos={0 0}, upload=207673]: Segment reached optimal threshold of 13 pieces. Dec 15 12:20:09.499212353 (236h51m9.370295398s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499223304 (236h51m9.370306349s) [noise=true , address=145.79.227.23:4546 , piece_id=GM62BNCRWAENFFAGRWCARMFSAQHGEJAHRIQOMAYAVSKKKT4U775Q, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499671397 (236h51m9.370754442s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34472->145.79.227.31:4797: use of closed network connection Dec 15 12:20:09.499682449 (236h51m9.370765484s) [noise=true , address=145.79.227.31:4797 , piece_id=GP4LKMSC4OKSGA2VPQPL2T2XC4NFVA32KEQIQUIZPHIUUGUQNVBQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499876678 (236h51m9.370959723s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 15 12:20:09.499886436 (236h51m9.370969491s) [noise=true , address=145.79.227.24:4583 , piece_id=Y3FMEWKXSVAP4MQIENDAHY37A54GATGM5HQ36VF5AGSFV2XO27RQ, seg_pos={0 0}, upload=207673]: Long tail cancellation due to optimal threshold Dec 15 12:20:09.499917546 (236h51m9.371000581s) [seg_pos={0 0}, upload=207673]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 15 12:20:09.500016104 (236h51m9.371099159s) [upload=207673]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 15 12:20:09.500051832 (236h51m9.371134867s) [upload=207673]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 15 12:20:09.898655888 (236h51m9.769738933s) [upload=207673]: Done uploading segments... Dec 15 12:20:09.898680846 (236h51m9.769763901s) [upload=207673]: Upload finished. err: <nil> Duration: 10.930594358s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)