Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................X....X................................] duration:
6.670890831s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:16:48 UTC)
orbiter warning: context deadline exceeded
Duration: 6.670908294s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXXX] duration:
5.001243915s trace:
https://jaeger.dev.storj.io/trace/165f5acbf3d4e0b7
FAIL (2025-12-30 13:16:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001266037s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration:
27.873454296s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:16:41 UTC)
orbiter warning: context deadline exceeded
Duration: 27.873475897s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.............X........X...........X..X........X...X...XXX...] duration:
26.410563601s trace:
https://jaeger.dev.storj.io/trace/ce900a8ac5e56de
FAIL (2025-12-30 13:16:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.410588069s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X..X.XX...............X..X.X..X....X........XX..............] duration:
6.410778106s trace:
https://jaeger.dev.storj.io/trace/4a02b772b0018a96
FAIL (2025-12-30 13:16:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded
======LOGS======
Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload
Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload
Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments...
Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false
Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment...
Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment.
Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment
Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments...
Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled
Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments...
Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil>
Duration: 6.410809488s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X...X.........X....X........X..............X.......X...X...] duration:
15.45569203s trace:
https://jaeger.dev.storj.io/trace/327137fdf99c8cd2
FAIL (2025-12-30 13:16:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
, piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463267094 (597h44m10.334350129s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463636957 (597h44m10.334719992s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463733021 (597h44m10.334816076s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463967257 (597h44m10.335050292s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464317554 (597h44m10.335400589s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464332772 (597h44m10.335415817s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464747913 (597h44m10.335830958s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464994803 (597h44m10.336077848s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.465113149 (597h44m10.336196184s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.465730814 (597h44m10.336813859s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.466479270 (597h44m10.337562315s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.466845497 (597h44m10.337928542s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.467111013 (597h44m10.338194058s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.467663886 (597h44m10.338746921s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:11.913537776 (597h44m11.784620821s) [upload=523949]: Next returned nil segment
Dec 30 13:13:11.913553607 (597h44m11.784636652s) [upload=523949]: Waiting for error group managing segments...
Dec 30 13:13:12.310633846 (597h44m12.181716891s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.369636326 (597h44m12.240719361s) [noise=true , address=145.79.227.5:4248 , piece_id=GVI3MNAX2S2H6IKO67GTHYD6Z2FXPDPNRRSFEBRICJKOZYOP6BLA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.489444639 (597h44m12.360527674s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.552298983 (597h44m12.423382018s) [noise=true , address=64.49.13.27:4967 , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.637395539 (597h44m12.508478574s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.647923005 (597h44m12.51900604s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.718170647 (597h44m12.589253692s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.731374770 (597h44m12.602457815s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.792347862 (597h44m12.663430917s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.202419249 (597h44m13.073502284s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.235840199 (597h44m13.106923234s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.322780525 (597h44m13.193863571s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.442893078 (597h44m13.313976113s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.631615082 (597h44m13.502698127s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.631640270 (597h44m13.502723315s) [seg_pos={0 0}, upload=523949]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:13:13.631902971 (597h44m13.502986036s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF): protocol: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection
Dec 30 13:13:13.631912879 (597h44m13.502995935s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632041715 (597h44m13.50312475s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 30 13:13:13.632057996 (597h44m13.503141031s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632602493 (597h44m13.503685528s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection
Dec 30 13:13:13.632610257 (597h44m13.503693302s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632858089 (597h44m13.503941134s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF
Dec 30 13:13:13.632864872 (597h44m13.503947907s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632959923 (597h44m13.504042958s) [seg_pos={0 0}, upload=523949]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:13:13.633117143 (597h44m13.504200188s) [upload=523949]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:13:13.633123475 (597h44m13.50420651s) [upload=523949]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:13:13.920616664 (597h44m13.791699709s) [upload=523949]: Done uploading segments...
Dec 30 13:13:13.920640600 (597h44m13.791723645s) [upload=523949]: Upload finished. err: <nil>
Duration: 15.455727436s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXXX] duration:
5.001732224s trace:
https://jaeger.dev.storj.io/trace/95bfbab3ecd0c11
FAIL (2025-12-30 13:15:45 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001757773s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [X.......................X....X..............................] duration:
6.711818997s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:15:17 UTC)
orbiter warning: context deadline exceeded
Duration: 6.711871466s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..............X........X...........X..X........X...X...XXX..] duration:
30.001467547s trace:
https://jaeger.dev.storj.io/trace/ce900a8ac5e56de
FAIL (2025-12-30 13:15:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001543883s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [X.X...X.........X....X........X..............X.......X...X..] duration:
30.017375859s trace:
https://jaeger.dev.storj.io/trace/327137fdf99c8cd2
FAIL (2025-12-30 13:15:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
, piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463267094 (597h44m10.334350129s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463636957 (597h44m10.334719992s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463733021 (597h44m10.334816076s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463967257 (597h44m10.335050292s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464317554 (597h44m10.335400589s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464332772 (597h44m10.335415817s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464747913 (597h44m10.335830958s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464994803 (597h44m10.336077848s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.465113149 (597h44m10.336196184s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.465730814 (597h44m10.336813859s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.466479270 (597h44m10.337562315s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.466845497 (597h44m10.337928542s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.467111013 (597h44m10.338194058s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.467663886 (597h44m10.338746921s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:11.913537776 (597h44m11.784620821s) [upload=523949]: Next returned nil segment
Dec 30 13:13:11.913553607 (597h44m11.784636652s) [upload=523949]: Waiting for error group managing segments...
Dec 30 13:13:12.310633846 (597h44m12.181716891s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.369636326 (597h44m12.240719361s) [noise=true , address=145.79.227.5:4248 , piece_id=GVI3MNAX2S2H6IKO67GTHYD6Z2FXPDPNRRSFEBRICJKOZYOP6BLA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.489444639 (597h44m12.360527674s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.552298983 (597h44m12.423382018s) [noise=true , address=64.49.13.27:4967 , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.637395539 (597h44m12.508478574s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.647923005 (597h44m12.51900604s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.718170647 (597h44m12.589253692s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.731374770 (597h44m12.602457815s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.792347862 (597h44m12.663430917s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.202419249 (597h44m13.073502284s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.235840199 (597h44m13.106923234s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.322780525 (597h44m13.193863571s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.442893078 (597h44m13.313976113s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.631615082 (597h44m13.502698127s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.631640270 (597h44m13.502723315s) [seg_pos={0 0}, upload=523949]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:13:13.631902971 (597h44m13.502986036s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF): protocol: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection
Dec 30 13:13:13.631912879 (597h44m13.502995935s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632041715 (597h44m13.50312475s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 30 13:13:13.632057996 (597h44m13.503141031s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632602493 (597h44m13.503685528s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection
Dec 30 13:13:13.632610257 (597h44m13.503693302s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632858089 (597h44m13.503941134s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF
Dec 30 13:13:13.632864872 (597h44m13.503947907s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632959923 (597h44m13.504042958s) [seg_pos={0 0}, upload=523949]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:13:13.633117143 (597h44m13.504200188s) [upload=523949]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:13:13.633123475 (597h44m13.50420651s) [upload=523949]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:13:13.920616664 (597h44m13.791699709s) [upload=523949]: Done uploading segments...
Dec 30 13:13:13.920640600 (597h44m13.791723645s) [upload=523949]: Upload finished. err: <nil>
Duration: 30.017403022s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXXX] duration:
5.001546741s trace:
https://jaeger.dev.storj.io/trace/a6028e6b925f91f
FAIL (2025-12-30 13:15:07 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001576957s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.X.......................X....X.............................] duration:
26.8105167s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:15:01 UTC)
orbiter warning: context deadline exceeded
Duration: 26.810548683s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X..X.XX...............X..X.X..X....X........XX.............] duration:
5.62349767s trace:
https://jaeger.dev.storj.io/trace/4a02b772b0018a96
FAIL (2025-12-30 13:14:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded
======LOGS======
Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload
Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload
Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments...
Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false
Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment...
Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment.
Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment
Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments...
Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled
Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments...
Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil>
Duration: 5.623529599s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.X.X...X.........X....X........X..............X.......X...X.] duration:
32.591698173s trace:
https://jaeger.dev.storj.io/trace/327137fdf99c8cd2
FAIL (2025-12-30 13:14:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
, piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463267094 (597h44m10.334350129s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463636957 (597h44m10.334719992s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463733021 (597h44m10.334816076s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463967257 (597h44m10.335050292s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464317554 (597h44m10.335400589s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464332772 (597h44m10.335415817s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464747913 (597h44m10.335830958s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464994803 (597h44m10.336077848s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.465113149 (597h44m10.336196184s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.465730814 (597h44m10.336813859s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.466479270 (597h44m10.337562315s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.466845497 (597h44m10.337928542s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.467111013 (597h44m10.338194058s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.467663886 (597h44m10.338746921s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:11.913537776 (597h44m11.784620821s) [upload=523949]: Next returned nil segment
Dec 30 13:13:11.913553607 (597h44m11.784636652s) [upload=523949]: Waiting for error group managing segments...
Dec 30 13:13:12.310633846 (597h44m12.181716891s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.369636326 (597h44m12.240719361s) [noise=true , address=145.79.227.5:4248 , piece_id=GVI3MNAX2S2H6IKO67GTHYD6Z2FXPDPNRRSFEBRICJKOZYOP6BLA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.489444639 (597h44m12.360527674s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.552298983 (597h44m12.423382018s) [noise=true , address=64.49.13.27:4967 , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.637395539 (597h44m12.508478574s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.647923005 (597h44m12.51900604s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.718170647 (597h44m12.589253692s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.731374770 (597h44m12.602457815s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.792347862 (597h44m12.663430917s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.202419249 (597h44m13.073502284s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.235840199 (597h44m13.106923234s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.322780525 (597h44m13.193863571s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.442893078 (597h44m13.313976113s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.631615082 (597h44m13.502698127s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.631640270 (597h44m13.502723315s) [seg_pos={0 0}, upload=523949]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:13:13.631902971 (597h44m13.502986036s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF): protocol: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection
Dec 30 13:13:13.631912879 (597h44m13.502995935s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632041715 (597h44m13.50312475s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 30 13:13:13.632057996 (597h44m13.503141031s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632602493 (597h44m13.503685528s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection
Dec 30 13:13:13.632610257 (597h44m13.503693302s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632858089 (597h44m13.503941134s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF
Dec 30 13:13:13.632864872 (597h44m13.503947907s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632959923 (597h44m13.504042958s) [seg_pos={0 0}, upload=523949]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:13:13.633117143 (597h44m13.504200188s) [upload=523949]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:13:13.633123475 (597h44m13.50420651s) [upload=523949]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:13:13.920616664 (597h44m13.791699709s) [upload=523949]: Done uploading segments...
Dec 30 13:13:13.920640600 (597h44m13.791723645s) [upload=523949]: Upload finished. err: <nil>
Duration: 32.591714694s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...............X........X...........X..X........X...X...XXX.] duration:
24.372656548s trace:
https://jaeger.dev.storj.io/trace/ce900a8ac5e56de
FAIL (2025-12-30 13:14:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.372900483s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..X.......................X....X............................] duration:
31.962208325s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:14:34 UTC)
orbiter warning: context deadline exceeded
Duration: 31.962232993s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..X..X.XX...............X..X.X..X....X........XX............] duration:
5.817348842s trace:
https://jaeger.dev.storj.io/trace/4a02b772b0018a96
FAIL (2025-12-30 13:14:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded
======LOGS======
Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload
Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload
Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments...
Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false
Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment...
Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment.
Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment
Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments...
Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled
Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments...
Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil>
Duration: 5.817374981s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [..X.X...X.........X....X........X..............X.......X...X] duration:
59.708755033s trace:
https://jaeger.dev.storj.io/trace/327137fdf99c8cd2
FAIL (2025-12-30 13:14:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
, piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463267094 (597h44m10.334350129s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463636957 (597h44m10.334719992s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463733021 (597h44m10.334816076s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.463967257 (597h44m10.335050292s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464317554 (597h44m10.335400589s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464332772 (597h44m10.335415817s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464747913 (597h44m10.335830958s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.464994803 (597h44m10.336077848s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.465113149 (597h44m10.336196184s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.465730814 (597h44m10.336813859s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.466479270 (597h44m10.337562315s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.466845497 (597h44m10.337928542s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.467111013 (597h44m10.338194058s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:10.467663886 (597h44m10.338746921s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Uploading piece...
Dec 30 13:13:11.913537776 (597h44m11.784620821s) [upload=523949]: Next returned nil segment
Dec 30 13:13:11.913553607 (597h44m11.784636652s) [upload=523949]: Waiting for error group managing segments...
Dec 30 13:13:12.310633846 (597h44m12.181716891s) [noise=true , address=145.79.227.18:4009 , piece_id=BYVA253BBGKUOK7INS3LEE3QSBRLLWMMJV5A2PZUQA2RBNWT4ZZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.369636326 (597h44m12.240719361s) [noise=true , address=145.79.227.5:4248 , piece_id=GVI3MNAX2S2H6IKO67GTHYD6Z2FXPDPNRRSFEBRICJKOZYOP6BLA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.489444639 (597h44m12.360527674s) [noise=true , address=145.79.227.49:4341 , piece_id=2MSWTVQQTLVFQYFGKELWL2JPYI7344J23GLHVUQUEGLCRDZGJVXA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.552298983 (597h44m12.423382018s) [noise=true , address=64.49.13.27:4967 , piece_id=UKK6CYYMVTBFF3PHDJ6LVR7VHVVMZZYO5PBNKRRXAAFKKGQPQZUA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.637395539 (597h44m12.508478574s) [noise=true , address=77.90.31.13:4649 , piece_id=RT6VXWG6TXFV3AJ3JHCPUAKI3KR3MSKVNPPJ3SSGECO4CRH546PQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.647923005 (597h44m12.51900604s) [noise=true , address=77.90.31.18:4834 , piece_id=A3KFC4KFDELIQRDLJOGRAYCNFC4EAXQL276C5ZMWXKN3YTOYI7ZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.718170647 (597h44m12.589253692s) [noise=true , address=77.90.31.21:4892 , piece_id=OT6ATUOEZDBN62GN6XXZF5F46OEJCPSZAZMKANKWYVIG3D6ZFMAQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.731374770 (597h44m12.602457815s) [noise=true , address=77.90.31.12:4623 , piece_id=QZI2VTB3D35QXNWGVZKSE7POFRZHBW3YJMEDH5SF6DXVTS6VRLSA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:12.792347862 (597h44m12.663430917s) [noise=true , address=45.91.116.10:4090 , piece_id=Q2ZBT5LPOMG4YS47AYUPAHUMAOPSQFDJFI7NXSIHI4GRF5K5W47A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.202419249 (597h44m13.073502284s) [noise=true , address=145.79.227.44:4208 , piece_id=EFOMVD7X3EGBRCNQDXZGIUPPRQVOEXCKNBESD45LVY3PWU6BTWZA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.235840199 (597h44m13.106923234s) [noise=true , address=145.79.227.30:4795 , piece_id=BTM46ZBI43EBMJQIUNCHCAB3SB3A3ICOSGFXN33DBSFSXLB5WBMQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.322780525 (597h44m13.193863571s) [noise=true , address=145.79.227.34:4932 , piece_id=3K67AT24AKS5N7C76GHYVYHIJNYSGDUKMIFH5TBQQGRYMDA2YXTQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.442893078 (597h44m13.313976113s) [noise=true , address=145.79.227.33:4889 , piece_id=K4IDNRUBBK7SMCM5KEP35FKJVPLF7DXWPXA2IAPFL66VL36MZPZQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.631615082 (597h44m13.502698127s) [noise=true , address=145.79.227.43:4160 , piece_id=CGLJK23URNZNPOYQWIYHCW7DILAFOSO6DVXXG44UQI3IZZCMNF4Q, seg_pos={0 0}, upload=523949]: Done uploading piece. err: <nil>
Dec 30 13:13:13.631640270 (597h44m13.502723315s) [seg_pos={0 0}, upload=523949]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:13:13.631902971 (597h44m13.502986036s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF): protocol: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->45.91.116.14:4211: use of closed network connection
Dec 30 13:13:13.631912879 (597h44m13.502995935s) [noise=true , address=45.91.116.14:4211 , piece_id=YOVPSR2U3TNJMURODDNZJHBBSO7LACWNUUDWULHVTCI5G4WRHY2A, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632041715 (597h44m13.50312475s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR): protocol: expected piece hash; context canceled; EOF
Dec 30 13:13:13.632057996 (597h44m13.503141031s) [noise=true , address=77.90.31.19:4870 , piece_id=J2EMSM3JLGXAZ7Q2BS2WCJJX4MAUNMCTSFCIB7577WQKMZHFLHEA, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632602493 (597h44m13.503685528s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57962->64.49.13.169:4706: use of closed network connection
Dec 30 13:13:13.632610257 (597h44m13.503693302s) [noise=true , address=64.49.13.169:4706 , piece_id=BIORMKATM22C4KNZCGQKPGGV5NRCHRUCZO55VVIOJWSOEAH2S7MQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632858089 (597h44m13.503941134s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF
Dec 30 13:13:13.632864872 (597h44m13.503947907s) [noise=true , address=77.90.31.10:4524 , piece_id=4ZA2JON2ATE67QQRB3QO5RNTXWPFP5GYYPBJJRRREBVV5SQ3WLNQ, seg_pos={0 0}, upload=523949]: Long tail cancellation due to optimal threshold
Dec 30 13:13:13.632959923 (597h44m13.504042958s) [seg_pos={0 0}, upload=523949]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:13:13.633117143 (597h44m13.504200188s) [upload=523949]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:13:13.633123475 (597h44m13.50420651s) [upload=523949]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:13:13.920616664 (597h44m13.791699709s) [upload=523949]: Done uploading segments...
Dec 30 13:13:13.920640600 (597h44m13.791723645s) [upload=523949]: Upload finished. err: <nil>
Duration: 59.70878446s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X..X..X.XX...............X..X.X..X....X........XX...........] duration:
7.129689254s trace:
https://jaeger.dev.storj.io/trace/4a02b772b0018a96
FAIL (2025-12-30 13:14:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded
======LOGS======
Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload
Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload
Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments...
Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false
Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment...
Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment.
Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment
Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments...
Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled
Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments...
Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil>
Duration: 7.129715675s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XXX] duration:
5.001947465s trace:
https://jaeger.dev.storj.io/trace/6e695ad6c1f26cc0
FAIL (2025-12-30 13:13:59 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001973263s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [................X........X...........X..X........X...X...XXX] duration:
30.001891421s trace:
https://jaeger.dev.storj.io/trace/ce900a8ac5e56de
FAIL (2025-12-30 13:13:54 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001914464s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.XX] duration:
5.001519392s trace:
https://jaeger.dev.storj.io/trace/18c02305d9c68469
FAIL (2025-12-30 13:13:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001543257s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...X.......................X....X...........................] duration:
29.760066518s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:13:18 UTC)
orbiter warning: context deadline exceeded
Duration: 29.760111965s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [XX..X..X.XX...............X..X.X..X....X........XX..........] duration:
6.057358829s trace:
https://jaeger.dev.storj.io/trace/4a02b772b0018a96
FAIL (2025-12-30 13:13:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded
======LOGS======
Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload
Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload
Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments...
Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false
Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment...
Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment.
Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment
Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments...
Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled
Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments...
Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil>
Duration: 6.057388897s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.................X........X...........X..X........X...X...XX] duration:
30.001072776s trace:
https://jaeger.dev.storj.io/trace/3e55fd7723e03982
FAIL (2025-12-30 13:13:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001084669s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [...X.X...X.........X....X........X..............X.......X...] duration:
27.103874289s trace:
https://jaeger.dev.storj.io/trace/6cf39a1e23e02046
FAIL (2025-12-30 13:12:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.13:4639 , piece_id=QXF6EDYISYL32GIMOCNKG6575QNQ6T3APX24JPM3DR3I7C6TNNIA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958032782 (597h41m37.829115827s) [noise=true , address=145.79.227.49:4353 , piece_id=QPAZHCFKI5T2J36OY34OEYGAGEEMIZOMLR3FG2IPQ7SS6SBPTDPA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958396123 (597h41m37.829479158s) [noise=true , address=145.79.227.30:4765 , piece_id=TBVK56UNGAMI2X5GSCXXGW5SYAZAF2BLRBIJMLAC4EIA7DNDGSNQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958713497 (597h41m37.829796542s) [noise=true , address=64.49.13.26:4949 , piece_id=6Q3U4H3YVKCGZV5KDD5FQJ7U2CF2ZR3U4ENOJZ27MS2WN25FNAQQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959011555 (597h41m37.8300946s) [noise=true , address=77.90.31.11:4578 , piece_id=KX4V7BOYIPDSGQPB56IVXLD7BBKIVEUJYGY5JTMFT3GEZAGVZFQA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959289034 (597h41m37.830372079s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959594836 (597h41m37.830677871s) [noise=true , address=193.149.164.143:4702 , piece_id=UPM47SHT32Q7UOL2HWWIAVEEXZM3O7SCGGYCIFSJAFYUO52H7NUQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959909225 (597h41m37.83099226s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.960889521 (597h41m37.831972556s) [noise=true , address=145.79.227.44:4206 , piece_id=TMLKWUBRSJP5AY4VDOIMRZAUATFDT6S565YKALCMHTXO5AV3JVUQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.961186145 (597h41m37.832269181s) [noise=true , address=77.90.31.14:4691 , piece_id=TZZLATMZMBDNTIJFP3RUSNST2DN2P5GEAXMVMYHU65QPIHS4F6WA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.962384538 (597h41m37.833467573s) [noise=true , address=45.91.116.11:4133 , piece_id=ZH56EDI6DKJA2PQNOWQANK5XUVAUEZRTKD46ZC7M3R4KL7QNLTOQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.963575245 (597h41m37.83465828s) [noise=true , address=145.79.227.15:4902 , piece_id=2VLDOU5ZQMRCZHMIGOKC4SJJOTGNSJJTY6TU5YWS7OENTWMI23HQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.964515666 (597h41m37.835598701s) [noise=true , address=77.90.31.12:4603 , piece_id=WXQYEEVJZYUICLPB3YZHNH7M72RGBROUWBFNE4QPTDNOTRBFZ4LQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.965640678 (597h41m37.836723713s) [noise=true , address=145.79.227.40:4061 , piece_id=HVY4N4K5NGOVKAK4MDLBWCUAA2HQDY2F7C2B22N33B24ZPB4XJHQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.967000438 (597h41m37.838083473s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.968064784 (597h41m37.839147819s) [noise=true , address=45.91.116.6:4979 , piece_id=NKKVUUGI62OETKOQK4E4FWONNG7C6IQVANDKD2FKWLA35LM6WAQQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:39.353701181 (597h41m39.224784216s) [upload=523911]: Next returned nil segment
Dec 30 13:10:39.353713886 (597h41m39.224796921s) [upload=523911]: Waiting for error group managing segments...
Dec 30 13:10:39.703948936 (597h41m39.575031981s) [noise=true , address=64.49.13.26:4949 , piece_id=6Q3U4H3YVKCGZV5KDD5FQJ7U2CF2ZR3U4ENOJZ27MS2WN25FNAQQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:39.833793920 (597h41m39.704876965s) [noise=true , address=145.79.227.15:4902 , piece_id=2VLDOU5ZQMRCZHMIGOKC4SJJOTGNSJJTY6TU5YWS7OENTWMI23HQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:39.855146559 (597h41m39.726229604s) [noise=true , address=145.79.227.7:4297 , piece_id=S4RYW7VDWIUUTUULXJ3IJWB5NTWKSMNITQL4FYO2DABNOAXXQBSA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.078817241 (597h41m39.949900276s) [noise=true , address=77.90.31.12:4603 , piece_id=WXQYEEVJZYUICLPB3YZHNH7M72RGBROUWBFNE4QPTDNOTRBFZ4LQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.112729124 (597h41m39.983812159s) [noise=true , address=77.90.31.11:4578 , piece_id=KX4V7BOYIPDSGQPB56IVXLD7BBKIVEUJYGY5JTMFT3GEZAGVZFQA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.114632028 (597h41m39.985715113s) [noise=true , address=77.90.31.14:4691 , piece_id=TZZLATMZMBDNTIJFP3RUSNST2DN2P5GEAXMVMYHU65QPIHS4F6WA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.155571372 (597h41m40.026654407s) [noise=true , address=45.91.116.11:4133 , piece_id=ZH56EDI6DKJA2PQNOWQANK5XUVAUEZRTKD46ZC7M3R4KL7QNLTOQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.209972590 (597h41m40.081055635s) [noise=true , address=45.91.116.6:4979 , piece_id=NKKVUUGI62OETKOQK4E4FWONNG7C6IQVANDKD2FKWLA35LM6WAQQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.233150905 (597h41m40.104233951s) [noise=true , address=77.90.31.13:4639 , piece_id=QXF6EDYISYL32GIMOCNKG6575QNQ6T3APX24JPM3DR3I7C6TNNIA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.706283119 (597h41m40.577366154s) [noise=true , address=145.79.227.44:4206 , piece_id=TMLKWUBRSJP5AY4VDOIMRZAUATFDT6S565YKALCMHTXO5AV3JVUQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.743670271 (597h41m40.614753306s) [noise=true , address=145.79.227.30:4765 , piece_id=TBVK56UNGAMI2X5GSCXXGW5SYAZAF2BLRBIJMLAC4EIA7DNDGSNQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.753163677 (597h41m40.624246712s) [noise=true , address=145.79.227.40:4061 , piece_id=HVY4N4K5NGOVKAK4MDLBWCUAA2HQDY2F7C2B22N33B24ZPB4XJHQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.848165074 (597h41m40.719248119s) [noise=true , address=145.79.227.49:4353 , piece_id=QPAZHCFKI5T2J36OY34OEYGAGEEMIZOMLR3FG2IPQ7SS6SBPTDPA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.866761605 (597h41m40.73784465s) [noise=true , address=193.149.164.143:4702 , piece_id=UPM47SHT32Q7UOL2HWWIAVEEXZM3O7SCGGYCIFSJAFYUO52H7NUQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.866785681 (597h41m40.737868726s) [seg_pos={0 0}, upload=523911]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:10:40.867011110 (597h41m40.738094155s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection
Dec 30 13:10:40.867021870 (597h41m40.738104905s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867270894 (597h41m40.738353929s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867317063 (597h41m40.738400098s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867355897 (597h41m40.738438942s) [noise=true , address=77.90.31.17:4779 , piece_id=YIM5DFXEX76WWSOINACHZSMWKN7FCYGA5B5I2NQJXOMPLULISJ3A, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867365996 (597h41m40.738449041s) [noise=true , address=77.90.31.17:4779 , piece_id=YIM5DFXEX76WWSOINACHZSMWKN7FCYGA5B5I2NQJXOMPLULISJ3A, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867611283 (597h41m40.738694318s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867643744 (597h41m40.73872678s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867665927 (597h41m40.738748962s) [seg_pos={0 0}, upload=523911]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:10:40.867828767 (597h41m40.738911803s) [upload=523911]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:10:40.867835520 (597h41m40.738918565s) [upload=523911]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:10:41.151013984 (597h41m41.022097029s) [upload=523911]: Done uploading segments...
Dec 30 13:10:41.151039152 (597h41m41.022122197s) [upload=523911]: Upload finished. err: <nil>
Duration: 27.103889437s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [..................X........X...........X..X........X...X...X] duration:
30.001358022s trace:
https://jaeger.dev.storj.io/trace/75c75e8d83db7aee
FAIL (2025-12-30 13:12:33 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001397706s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [....X.X...X.........X....X........X..............X.......X..] duration:
31.197562095s trace:
https://jaeger.dev.storj.io/trace/6cf39a1e23e02046
FAIL (2025-12-30 13:12:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.13:4639 , piece_id=QXF6EDYISYL32GIMOCNKG6575QNQ6T3APX24JPM3DR3I7C6TNNIA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958032782 (597h41m37.829115827s) [noise=true , address=145.79.227.49:4353 , piece_id=QPAZHCFKI5T2J36OY34OEYGAGEEMIZOMLR3FG2IPQ7SS6SBPTDPA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958396123 (597h41m37.829479158s) [noise=true , address=145.79.227.30:4765 , piece_id=TBVK56UNGAMI2X5GSCXXGW5SYAZAF2BLRBIJMLAC4EIA7DNDGSNQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958713497 (597h41m37.829796542s) [noise=true , address=64.49.13.26:4949 , piece_id=6Q3U4H3YVKCGZV5KDD5FQJ7U2CF2ZR3U4ENOJZ27MS2WN25FNAQQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959011555 (597h41m37.8300946s) [noise=true , address=77.90.31.11:4578 , piece_id=KX4V7BOYIPDSGQPB56IVXLD7BBKIVEUJYGY5JTMFT3GEZAGVZFQA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959289034 (597h41m37.830372079s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959594836 (597h41m37.830677871s) [noise=true , address=193.149.164.143:4702 , piece_id=UPM47SHT32Q7UOL2HWWIAVEEXZM3O7SCGGYCIFSJAFYUO52H7NUQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959909225 (597h41m37.83099226s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.960889521 (597h41m37.831972556s) [noise=true , address=145.79.227.44:4206 , piece_id=TMLKWUBRSJP5AY4VDOIMRZAUATFDT6S565YKALCMHTXO5AV3JVUQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.961186145 (597h41m37.832269181s) [noise=true , address=77.90.31.14:4691 , piece_id=TZZLATMZMBDNTIJFP3RUSNST2DN2P5GEAXMVMYHU65QPIHS4F6WA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.962384538 (597h41m37.833467573s) [noise=true , address=45.91.116.11:4133 , piece_id=ZH56EDI6DKJA2PQNOWQANK5XUVAUEZRTKD46ZC7M3R4KL7QNLTOQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.963575245 (597h41m37.83465828s) [noise=true , address=145.79.227.15:4902 , piece_id=2VLDOU5ZQMRCZHMIGOKC4SJJOTGNSJJTY6TU5YWS7OENTWMI23HQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.964515666 (597h41m37.835598701s) [noise=true , address=77.90.31.12:4603 , piece_id=WXQYEEVJZYUICLPB3YZHNH7M72RGBROUWBFNE4QPTDNOTRBFZ4LQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.965640678 (597h41m37.836723713s) [noise=true , address=145.79.227.40:4061 , piece_id=HVY4N4K5NGOVKAK4MDLBWCUAA2HQDY2F7C2B22N33B24ZPB4XJHQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.967000438 (597h41m37.838083473s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.968064784 (597h41m37.839147819s) [noise=true , address=45.91.116.6:4979 , piece_id=NKKVUUGI62OETKOQK4E4FWONNG7C6IQVANDKD2FKWLA35LM6WAQQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:39.353701181 (597h41m39.224784216s) [upload=523911]: Next returned nil segment
Dec 30 13:10:39.353713886 (597h41m39.224796921s) [upload=523911]: Waiting for error group managing segments...
Dec 30 13:10:39.703948936 (597h41m39.575031981s) [noise=true , address=64.49.13.26:4949 , piece_id=6Q3U4H3YVKCGZV5KDD5FQJ7U2CF2ZR3U4ENOJZ27MS2WN25FNAQQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:39.833793920 (597h41m39.704876965s) [noise=true , address=145.79.227.15:4902 , piece_id=2VLDOU5ZQMRCZHMIGOKC4SJJOTGNSJJTY6TU5YWS7OENTWMI23HQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:39.855146559 (597h41m39.726229604s) [noise=true , address=145.79.227.7:4297 , piece_id=S4RYW7VDWIUUTUULXJ3IJWB5NTWKSMNITQL4FYO2DABNOAXXQBSA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.078817241 (597h41m39.949900276s) [noise=true , address=77.90.31.12:4603 , piece_id=WXQYEEVJZYUICLPB3YZHNH7M72RGBROUWBFNE4QPTDNOTRBFZ4LQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.112729124 (597h41m39.983812159s) [noise=true , address=77.90.31.11:4578 , piece_id=KX4V7BOYIPDSGQPB56IVXLD7BBKIVEUJYGY5JTMFT3GEZAGVZFQA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.114632028 (597h41m39.985715113s) [noise=true , address=77.90.31.14:4691 , piece_id=TZZLATMZMBDNTIJFP3RUSNST2DN2P5GEAXMVMYHU65QPIHS4F6WA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.155571372 (597h41m40.026654407s) [noise=true , address=45.91.116.11:4133 , piece_id=ZH56EDI6DKJA2PQNOWQANK5XUVAUEZRTKD46ZC7M3R4KL7QNLTOQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.209972590 (597h41m40.081055635s) [noise=true , address=45.91.116.6:4979 , piece_id=NKKVUUGI62OETKOQK4E4FWONNG7C6IQVANDKD2FKWLA35LM6WAQQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.233150905 (597h41m40.104233951s) [noise=true , address=77.90.31.13:4639 , piece_id=QXF6EDYISYL32GIMOCNKG6575QNQ6T3APX24JPM3DR3I7C6TNNIA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.706283119 (597h41m40.577366154s) [noise=true , address=145.79.227.44:4206 , piece_id=TMLKWUBRSJP5AY4VDOIMRZAUATFDT6S565YKALCMHTXO5AV3JVUQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.743670271 (597h41m40.614753306s) [noise=true , address=145.79.227.30:4765 , piece_id=TBVK56UNGAMI2X5GSCXXGW5SYAZAF2BLRBIJMLAC4EIA7DNDGSNQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.753163677 (597h41m40.624246712s) [noise=true , address=145.79.227.40:4061 , piece_id=HVY4N4K5NGOVKAK4MDLBWCUAA2HQDY2F7C2B22N33B24ZPB4XJHQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.848165074 (597h41m40.719248119s) [noise=true , address=145.79.227.49:4353 , piece_id=QPAZHCFKI5T2J36OY34OEYGAGEEMIZOMLR3FG2IPQ7SS6SBPTDPA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.866761605 (597h41m40.73784465s) [noise=true , address=193.149.164.143:4702 , piece_id=UPM47SHT32Q7UOL2HWWIAVEEXZM3O7SCGGYCIFSJAFYUO52H7NUQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.866785681 (597h41m40.737868726s) [seg_pos={0 0}, upload=523911]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:10:40.867011110 (597h41m40.738094155s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection
Dec 30 13:10:40.867021870 (597h41m40.738104905s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867270894 (597h41m40.738353929s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867317063 (597h41m40.738400098s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867355897 (597h41m40.738438942s) [noise=true , address=77.90.31.17:4779 , piece_id=YIM5DFXEX76WWSOINACHZSMWKN7FCYGA5B5I2NQJXOMPLULISJ3A, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867365996 (597h41m40.738449041s) [noise=true , address=77.90.31.17:4779 , piece_id=YIM5DFXEX76WWSOINACHZSMWKN7FCYGA5B5I2NQJXOMPLULISJ3A, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867611283 (597h41m40.738694318s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867643744 (597h41m40.73872678s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867665927 (597h41m40.738748962s) [seg_pos={0 0}, upload=523911]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:10:40.867828767 (597h41m40.738911803s) [upload=523911]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:10:40.867835520 (597h41m40.738918565s) [upload=523911]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:10:41.151013984 (597h41m41.022097029s) [upload=523911]: Done uploading segments...
Dec 30 13:10:41.151039152 (597h41m41.022122197s) [upload=523911]: Upload finished. err: <nil>
Duration: 31.197588315s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.X] duration:
5.001465237s trace:
https://jaeger.dev.storj.io/trace/70dc7699eb1ff901
FAIL (2025-12-30 13:12:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001489594s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....X.......................X....X..........................] duration:
23.234854285s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:12:22 UTC)
orbiter warning: context deadline exceeded
Duration: 23.234883511s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.XX..X..X.XX...............X..X.X..X....X........XX.........] duration:
5.919388019s trace:
https://jaeger.dev.storj.io/trace/4a02b772b0018a96
FAIL (2025-12-30 13:12:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded
======LOGS======
Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload
Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload
Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments...
Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false
Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment...
Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment.
Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment
Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments...
Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled
Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments...
Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil>
Duration: 5.919420721s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX.] duration:
3.91902999s trace:
https://jaeger.dev.storj.io/trace/22a691989a420ea5
FAIL (2025-12-30 13:11:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.919046892s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [.....X.X...X.........X....X........X..............X.......X.] duration:
15.339853442s trace:
https://jaeger.dev.storj.io/trace/6cf39a1e23e02046
FAIL (2025-12-30 13:11:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.13:4639 , piece_id=QXF6EDYISYL32GIMOCNKG6575QNQ6T3APX24JPM3DR3I7C6TNNIA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958032782 (597h41m37.829115827s) [noise=true , address=145.79.227.49:4353 , piece_id=QPAZHCFKI5T2J36OY34OEYGAGEEMIZOMLR3FG2IPQ7SS6SBPTDPA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958396123 (597h41m37.829479158s) [noise=true , address=145.79.227.30:4765 , piece_id=TBVK56UNGAMI2X5GSCXXGW5SYAZAF2BLRBIJMLAC4EIA7DNDGSNQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958713497 (597h41m37.829796542s) [noise=true , address=64.49.13.26:4949 , piece_id=6Q3U4H3YVKCGZV5KDD5FQJ7U2CF2ZR3U4ENOJZ27MS2WN25FNAQQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959011555 (597h41m37.8300946s) [noise=true , address=77.90.31.11:4578 , piece_id=KX4V7BOYIPDSGQPB56IVXLD7BBKIVEUJYGY5JTMFT3GEZAGVZFQA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959289034 (597h41m37.830372079s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959594836 (597h41m37.830677871s) [noise=true , address=193.149.164.143:4702 , piece_id=UPM47SHT32Q7UOL2HWWIAVEEXZM3O7SCGGYCIFSJAFYUO52H7NUQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959909225 (597h41m37.83099226s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.960889521 (597h41m37.831972556s) [noise=true , address=145.79.227.44:4206 , piece_id=TMLKWUBRSJP5AY4VDOIMRZAUATFDT6S565YKALCMHTXO5AV3JVUQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.961186145 (597h41m37.832269181s) [noise=true , address=77.90.31.14:4691 , piece_id=TZZLATMZMBDNTIJFP3RUSNST2DN2P5GEAXMVMYHU65QPIHS4F6WA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.962384538 (597h41m37.833467573s) [noise=true , address=45.91.116.11:4133 , piece_id=ZH56EDI6DKJA2PQNOWQANK5XUVAUEZRTKD46ZC7M3R4KL7QNLTOQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.963575245 (597h41m37.83465828s) [noise=true , address=145.79.227.15:4902 , piece_id=2VLDOU5ZQMRCZHMIGOKC4SJJOTGNSJJTY6TU5YWS7OENTWMI23HQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.964515666 (597h41m37.835598701s) [noise=true , address=77.90.31.12:4603 , piece_id=WXQYEEVJZYUICLPB3YZHNH7M72RGBROUWBFNE4QPTDNOTRBFZ4LQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.965640678 (597h41m37.836723713s) [noise=true , address=145.79.227.40:4061 , piece_id=HVY4N4K5NGOVKAK4MDLBWCUAA2HQDY2F7C2B22N33B24ZPB4XJHQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.967000438 (597h41m37.838083473s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.968064784 (597h41m37.839147819s) [noise=true , address=45.91.116.6:4979 , piece_id=NKKVUUGI62OETKOQK4E4FWONNG7C6IQVANDKD2FKWLA35LM6WAQQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:39.353701181 (597h41m39.224784216s) [upload=523911]: Next returned nil segment
Dec 30 13:10:39.353713886 (597h41m39.224796921s) [upload=523911]: Waiting for error group managing segments...
Dec 30 13:10:39.703948936 (597h41m39.575031981s) [noise=true , address=64.49.13.26:4949 , piece_id=6Q3U4H3YVKCGZV5KDD5FQJ7U2CF2ZR3U4ENOJZ27MS2WN25FNAQQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:39.833793920 (597h41m39.704876965s) [noise=true , address=145.79.227.15:4902 , piece_id=2VLDOU5ZQMRCZHMIGOKC4SJJOTGNSJJTY6TU5YWS7OENTWMI23HQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:39.855146559 (597h41m39.726229604s) [noise=true , address=145.79.227.7:4297 , piece_id=S4RYW7VDWIUUTUULXJ3IJWB5NTWKSMNITQL4FYO2DABNOAXXQBSA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.078817241 (597h41m39.949900276s) [noise=true , address=77.90.31.12:4603 , piece_id=WXQYEEVJZYUICLPB3YZHNH7M72RGBROUWBFNE4QPTDNOTRBFZ4LQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.112729124 (597h41m39.983812159s) [noise=true , address=77.90.31.11:4578 , piece_id=KX4V7BOYIPDSGQPB56IVXLD7BBKIVEUJYGY5JTMFT3GEZAGVZFQA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.114632028 (597h41m39.985715113s) [noise=true , address=77.90.31.14:4691 , piece_id=TZZLATMZMBDNTIJFP3RUSNST2DN2P5GEAXMVMYHU65QPIHS4F6WA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.155571372 (597h41m40.026654407s) [noise=true , address=45.91.116.11:4133 , piece_id=ZH56EDI6DKJA2PQNOWQANK5XUVAUEZRTKD46ZC7M3R4KL7QNLTOQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.209972590 (597h41m40.081055635s) [noise=true , address=45.91.116.6:4979 , piece_id=NKKVUUGI62OETKOQK4E4FWONNG7C6IQVANDKD2FKWLA35LM6WAQQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.233150905 (597h41m40.104233951s) [noise=true , address=77.90.31.13:4639 , piece_id=QXF6EDYISYL32GIMOCNKG6575QNQ6T3APX24JPM3DR3I7C6TNNIA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.706283119 (597h41m40.577366154s) [noise=true , address=145.79.227.44:4206 , piece_id=TMLKWUBRSJP5AY4VDOIMRZAUATFDT6S565YKALCMHTXO5AV3JVUQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.743670271 (597h41m40.614753306s) [noise=true , address=145.79.227.30:4765 , piece_id=TBVK56UNGAMI2X5GSCXXGW5SYAZAF2BLRBIJMLAC4EIA7DNDGSNQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.753163677 (597h41m40.624246712s) [noise=true , address=145.79.227.40:4061 , piece_id=HVY4N4K5NGOVKAK4MDLBWCUAA2HQDY2F7C2B22N33B24ZPB4XJHQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.848165074 (597h41m40.719248119s) [noise=true , address=145.79.227.49:4353 , piece_id=QPAZHCFKI5T2J36OY34OEYGAGEEMIZOMLR3FG2IPQ7SS6SBPTDPA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.866761605 (597h41m40.73784465s) [noise=true , address=193.149.164.143:4702 , piece_id=UPM47SHT32Q7UOL2HWWIAVEEXZM3O7SCGGYCIFSJAFYUO52H7NUQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.866785681 (597h41m40.737868726s) [seg_pos={0 0}, upload=523911]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:10:40.867011110 (597h41m40.738094155s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection
Dec 30 13:10:40.867021870 (597h41m40.738104905s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867270894 (597h41m40.738353929s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867317063 (597h41m40.738400098s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867355897 (597h41m40.738438942s) [noise=true , address=77.90.31.17:4779 , piece_id=YIM5DFXEX76WWSOINACHZSMWKN7FCYGA5B5I2NQJXOMPLULISJ3A, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867365996 (597h41m40.738449041s) [noise=true , address=77.90.31.17:4779 , piece_id=YIM5DFXEX76WWSOINACHZSMWKN7FCYGA5B5I2NQJXOMPLULISJ3A, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867611283 (597h41m40.738694318s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867643744 (597h41m40.73872678s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867665927 (597h41m40.738748962s) [seg_pos={0 0}, upload=523911]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:10:40.867828767 (597h41m40.738911803s) [upload=523911]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:10:40.867835520 (597h41m40.738918565s) [upload=523911]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:10:41.151013984 (597h41m41.022097029s) [upload=523911]: Done uploading segments...
Dec 30 13:10:41.151039152 (597h41m41.022122197s) [upload=523911]: Upload finished. err: <nil>
Duration: 15.339868242s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...................X........X...........X..X........X...X...] duration:
18.389251193s trace:
https://jaeger.dev.storj.io/trace/1bcf024bfb319295
FAIL (2025-12-30 13:11:45 UTC)
orbiter warning: context deadline exceeded
Duration: 18.389277222s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [X.....X.X...X.........X....X........X..............X.......X] duration:
1m0.004252024s trace:
https://jaeger.dev.storj.io/trace/6cf39a1e23e02046
FAIL (2025-12-30 13:11:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
[noise=true , address=77.90.31.13:4639 , piece_id=QXF6EDYISYL32GIMOCNKG6575QNQ6T3APX24JPM3DR3I7C6TNNIA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958032782 (597h41m37.829115827s) [noise=true , address=145.79.227.49:4353 , piece_id=QPAZHCFKI5T2J36OY34OEYGAGEEMIZOMLR3FG2IPQ7SS6SBPTDPA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958396123 (597h41m37.829479158s) [noise=true , address=145.79.227.30:4765 , piece_id=TBVK56UNGAMI2X5GSCXXGW5SYAZAF2BLRBIJMLAC4EIA7DNDGSNQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.958713497 (597h41m37.829796542s) [noise=true , address=64.49.13.26:4949 , piece_id=6Q3U4H3YVKCGZV5KDD5FQJ7U2CF2ZR3U4ENOJZ27MS2WN25FNAQQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959011555 (597h41m37.8300946s) [noise=true , address=77.90.31.11:4578 , piece_id=KX4V7BOYIPDSGQPB56IVXLD7BBKIVEUJYGY5JTMFT3GEZAGVZFQA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959289034 (597h41m37.830372079s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959594836 (597h41m37.830677871s) [noise=true , address=193.149.164.143:4702 , piece_id=UPM47SHT32Q7UOL2HWWIAVEEXZM3O7SCGGYCIFSJAFYUO52H7NUQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.959909225 (597h41m37.83099226s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.960889521 (597h41m37.831972556s) [noise=true , address=145.79.227.44:4206 , piece_id=TMLKWUBRSJP5AY4VDOIMRZAUATFDT6S565YKALCMHTXO5AV3JVUQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.961186145 (597h41m37.832269181s) [noise=true , address=77.90.31.14:4691 , piece_id=TZZLATMZMBDNTIJFP3RUSNST2DN2P5GEAXMVMYHU65QPIHS4F6WA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.962384538 (597h41m37.833467573s) [noise=true , address=45.91.116.11:4133 , piece_id=ZH56EDI6DKJA2PQNOWQANK5XUVAUEZRTKD46ZC7M3R4KL7QNLTOQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.963575245 (597h41m37.83465828s) [noise=true , address=145.79.227.15:4902 , piece_id=2VLDOU5ZQMRCZHMIGOKC4SJJOTGNSJJTY6TU5YWS7OENTWMI23HQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.964515666 (597h41m37.835598701s) [noise=true , address=77.90.31.12:4603 , piece_id=WXQYEEVJZYUICLPB3YZHNH7M72RGBROUWBFNE4QPTDNOTRBFZ4LQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.965640678 (597h41m37.836723713s) [noise=true , address=145.79.227.40:4061 , piece_id=HVY4N4K5NGOVKAK4MDLBWCUAA2HQDY2F7C2B22N33B24ZPB4XJHQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.967000438 (597h41m37.838083473s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:37.968064784 (597h41m37.839147819s) [noise=true , address=45.91.116.6:4979 , piece_id=NKKVUUGI62OETKOQK4E4FWONNG7C6IQVANDKD2FKWLA35LM6WAQQ, seg_pos={0 0}, upload=523911]: Uploading piece...
Dec 30 13:10:39.353701181 (597h41m39.224784216s) [upload=523911]: Next returned nil segment
Dec 30 13:10:39.353713886 (597h41m39.224796921s) [upload=523911]: Waiting for error group managing segments...
Dec 30 13:10:39.703948936 (597h41m39.575031981s) [noise=true , address=64.49.13.26:4949 , piece_id=6Q3U4H3YVKCGZV5KDD5FQJ7U2CF2ZR3U4ENOJZ27MS2WN25FNAQQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:39.833793920 (597h41m39.704876965s) [noise=true , address=145.79.227.15:4902 , piece_id=2VLDOU5ZQMRCZHMIGOKC4SJJOTGNSJJTY6TU5YWS7OENTWMI23HQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:39.855146559 (597h41m39.726229604s) [noise=true , address=145.79.227.7:4297 , piece_id=S4RYW7VDWIUUTUULXJ3IJWB5NTWKSMNITQL4FYO2DABNOAXXQBSA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.078817241 (597h41m39.949900276s) [noise=true , address=77.90.31.12:4603 , piece_id=WXQYEEVJZYUICLPB3YZHNH7M72RGBROUWBFNE4QPTDNOTRBFZ4LQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.112729124 (597h41m39.983812159s) [noise=true , address=77.90.31.11:4578 , piece_id=KX4V7BOYIPDSGQPB56IVXLD7BBKIVEUJYGY5JTMFT3GEZAGVZFQA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.114632028 (597h41m39.985715113s) [noise=true , address=77.90.31.14:4691 , piece_id=TZZLATMZMBDNTIJFP3RUSNST2DN2P5GEAXMVMYHU65QPIHS4F6WA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.155571372 (597h41m40.026654407s) [noise=true , address=45.91.116.11:4133 , piece_id=ZH56EDI6DKJA2PQNOWQANK5XUVAUEZRTKD46ZC7M3R4KL7QNLTOQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.209972590 (597h41m40.081055635s) [noise=true , address=45.91.116.6:4979 , piece_id=NKKVUUGI62OETKOQK4E4FWONNG7C6IQVANDKD2FKWLA35LM6WAQQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.233150905 (597h41m40.104233951s) [noise=true , address=77.90.31.13:4639 , piece_id=QXF6EDYISYL32GIMOCNKG6575QNQ6T3APX24JPM3DR3I7C6TNNIA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.706283119 (597h41m40.577366154s) [noise=true , address=145.79.227.44:4206 , piece_id=TMLKWUBRSJP5AY4VDOIMRZAUATFDT6S565YKALCMHTXO5AV3JVUQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.743670271 (597h41m40.614753306s) [noise=true , address=145.79.227.30:4765 , piece_id=TBVK56UNGAMI2X5GSCXXGW5SYAZAF2BLRBIJMLAC4EIA7DNDGSNQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.753163677 (597h41m40.624246712s) [noise=true , address=145.79.227.40:4061 , piece_id=HVY4N4K5NGOVKAK4MDLBWCUAA2HQDY2F7C2B22N33B24ZPB4XJHQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.848165074 (597h41m40.719248119s) [noise=true , address=145.79.227.49:4353 , piece_id=QPAZHCFKI5T2J36OY34OEYGAGEEMIZOMLR3FG2IPQ7SS6SBPTDPA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.866761605 (597h41m40.73784465s) [noise=true , address=193.149.164.143:4702 , piece_id=UPM47SHT32Q7UOL2HWWIAVEEXZM3O7SCGGYCIFSJAFYUO52H7NUQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: <nil>
Dec 30 13:10:40.866785681 (597h41m40.737868726s) [seg_pos={0 0}, upload=523911]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:10:40.867011110 (597h41m40.738094155s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48642->64.49.13.169:4715: use of closed network connection
Dec 30 13:10:40.867021870 (597h41m40.738104905s) [noise=true , address=64.49.13.169:4715 , piece_id=MJLAHVICRL5ZD3LCWO4H3SWYK35RBKINTIFZHFQTYI3O3HPDUBOA, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867270894 (597h41m40.738353929s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867317063 (597h41m40.738400098s) [noise=true , address=77.90.31.10:4533 , piece_id=JC5CUGD6FEP2TBWIGE34VYUXW3H4ZNGDHWDWPVK6GCH4MZGJRQPA, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867355897 (597h41m40.738438942s) [noise=true , address=77.90.31.17:4779 , piece_id=YIM5DFXEX76WWSOINACHZSMWKN7FCYGA5B5I2NQJXOMPLULISJ3A, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HfiiWLGzNiY2CiGyBUWZ11t1sRY4duPLvAMdfgi4jMetx2ePN): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867365996 (597h41m40.738449041s) [noise=true , address=77.90.31.17:4779 , piece_id=YIM5DFXEX76WWSOINACHZSMWKN7FCYGA5B5I2NQJXOMPLULISJ3A, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867611283 (597h41m40.738694318s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF
Dec 30 13:10:40.867643744 (597h41m40.73872678s) [noise=true , address=145.79.227.35:4490 , piece_id=4NVFG4QEKRXQTD7GWVO65GHUZHGGTVY6X4QVOYSBEJ5E5SKMOUNQ, seg_pos={0 0}, upload=523911]: Long tail cancellation due to optimal threshold
Dec 30 13:10:40.867665927 (597h41m40.738748962s) [seg_pos={0 0}, upload=523911]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:10:40.867828767 (597h41m40.738911803s) [upload=523911]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:10:40.867835520 (597h41m40.738918565s) [upload=523911]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:10:41.151013984 (597h41m41.022097029s) [upload=523911]: Done uploading segments...
Dec 30 13:10:41.151039152 (597h41m41.022122197s) [upload=523911]: Upload finished. err: <nil>
Duration: 1m0.00426606s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X.XX..X..X.XX...............X..X.X..X....X........XX........] duration:
5.894820664s trace:
https://jaeger.dev.storj.io/trace/4a02b772b0018a96
FAIL (2025-12-30 13:11:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded
======LOGS======
Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload
Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload
Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments...
Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false
Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment...
Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment.
Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment
Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments...
Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled
Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments...
Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil>
Duration: 5.894846544s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXXX] duration:
5.001884593s trace:
https://jaeger.dev.storj.io/trace/22a691989a420ea5
FAIL (2025-12-30 13:11:22 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00195632s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....................X........X...........X..X........X...X..] duration:
18.159146005s trace:
https://jaeger.dev.storj.io/trace/1bcf024bfb319295
FAIL (2025-12-30 13:11:20 UTC)
orbiter warning: context deadline exceeded
Duration: 18.159173096s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....X.......................X....X.........................] duration:
18.15022525s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:11:15 UTC)
orbiter warning: context deadline exceeded
Duration: 18.150252762s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....................X........X...........X..X........X...X.] duration:
24.797738867s trace:
https://jaeger.dev.storj.io/trace/1bcf024bfb319295
FAIL (2025-12-30 13:10:50 UTC)
orbiter warning: context deadline exceeded
Duration: 24.797769005s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [.X.....X.X...X.........X....X........X..............X.......] duration:
45.182361611s trace:
https://jaeger.dev.storj.io/trace/59feccf4c7c3ba3a
FAIL (2025-12-30 13:10:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
ec 30 13:04:06.775086282 (597h35m6.646169327s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.775516872 (597h35m6.646599907s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.775911253 (597h35m6.646994298s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.776366340 (597h35m6.647449395s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.777866967 (597h35m6.648950012s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.778495334 (597h35m6.649578389s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.779041454 (597h35m6.650124499s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.779558509 (597h35m6.650641554s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.780053321 (597h35m6.651136366s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.780550918 (597h35m6.651633963s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.781069085 (597h35m6.65215213s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.781978246 (597h35m6.653061291s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.782926150 (597h35m6.654009195s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.783851333 (597h35m6.654934388s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.784805539 (597h35m6.655888584s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.786955854 (597h35m6.658038899s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:08.223121485 (597h35m8.09420453s) [upload=523818]: Next returned nil segment
Dec 30 13:04:08.223139148 (597h35m8.094222193s) [upload=523818]: Waiting for error group managing segments...
Dec 30 13:04:08.717719473 (597h35m8.588802518s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.717831707 (597h35m8.588914752s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.743536652 (597h35m8.614619687s) [noise=true , address=145.79.227.43:4187 , piece_id=NQCXR5HGRYEQ3WW27PQJLMUOEEUXZDGGQJQQO5NYGHPXPAQFWG7A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.848591082 (597h35m8.719674127s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.899414969 (597h35m8.770498014s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.123129336 (597h35m8.994212371s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.211969381 (597h35m9.083052416s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.294556076 (597h35m9.165639142s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.479068837 (597h35m9.350151872s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.510445983 (597h35m9.381529018s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.536218769 (597h35m9.407301814s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.566859122 (597h35m9.437942167s) [noise=true , address=145.79.227.33:4877 , piece_id=BMMH4UJO7OQRDWK4QSSNCA3FVEUYXTX3TPWO62WMUG6DMQ5VTBEA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.619658661 (597h35m9.490741696s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.621672355 (597h35m9.49275539s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.621683898 (597h35m9.492766943s) [seg_pos={0 0}, upload=523818]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:04:09.621901051 (597h35m9.492984086s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: expected piece hash; context canceled; EOF
Dec 30 13:04:09.621910259 (597h35m9.492993294s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622063460 (597h35m9.493146505s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection
Dec 30 13:04:09.622087205 (597h35m9.49317025s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622135457 (597h35m9.493218492s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF
Dec 30 13:04:09.622142100 (597h35m9.493225135s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622242572 (597h35m9.493325607s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF
Dec 30 13:04:09.622247882 (597h35m9.493330927s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622337151 (597h35m9.493420186s) [seg_pos={0 0}, upload=523818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:04:09.622456739 (597h35m9.493539795s) [upload=523818]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:04:09.622466388 (597h35m9.493549433s) [upload=523818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:04:09.936998074 (597h35m9.808081119s) [upload=523818]: Done uploading segments...
Dec 30 13:04:09.937018172 (597h35m9.808101217s) [upload=523818]: Upload finished. err: <nil>
Duration: 45.182392038s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X.XX..X..X.XX...............X..X.X..X....X........XX.......] duration:
5.802954543s trace:
https://jaeger.dev.storj.io/trace/4a02b772b0018a96
FAIL (2025-12-30 13:10:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded
======LOGS======
Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload
Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload
Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments...
Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false
Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment...
Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment.
Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment
Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments...
Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled
Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments...
Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil>
Duration: 5.802973179s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......X.......................X....X........................] duration:
26.931411571s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:10:29 UTC)
orbiter warning: context deadline exceeded
Duration: 26.931442238s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XXX] duration:
5.001145799s trace:
https://jaeger.dev.storj.io/trace/4233fb50954abbc9
FAIL (2025-12-30 13:10:20 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00117344s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [.X.XXXXXXXXXXX.X.XXXXXXXXX.X.XXXXXX.X.XXX.XXXX.XXXXXXXX...XX] duration:
5.001517679s trace:
https://jaeger.dev.storj.io/trace/5e74163fb7ce0ef9
FAIL (2025-12-30 13:09:57 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001550513s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................X........X...........X..X........X...X] duration:
30.001353822s trace:
https://jaeger.dev.storj.io/trace/1bcf024bfb319295
FAIL (2025-12-30 13:09:55 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001378729s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [..X.....X.X...X.........X....X........X..............X......] duration:
31.231396925s trace:
https://jaeger.dev.storj.io/trace/59feccf4c7c3ba3a
FAIL (2025-12-30 13:09:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
ec 30 13:04:06.775086282 (597h35m6.646169327s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.775516872 (597h35m6.646599907s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.775911253 (597h35m6.646994298s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.776366340 (597h35m6.647449395s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.777866967 (597h35m6.648950012s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.778495334 (597h35m6.649578389s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.779041454 (597h35m6.650124499s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.779558509 (597h35m6.650641554s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.780053321 (597h35m6.651136366s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.780550918 (597h35m6.651633963s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.781069085 (597h35m6.65215213s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.781978246 (597h35m6.653061291s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.782926150 (597h35m6.654009195s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.783851333 (597h35m6.654934388s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.784805539 (597h35m6.655888584s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.786955854 (597h35m6.658038899s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:08.223121485 (597h35m8.09420453s) [upload=523818]: Next returned nil segment
Dec 30 13:04:08.223139148 (597h35m8.094222193s) [upload=523818]: Waiting for error group managing segments...
Dec 30 13:04:08.717719473 (597h35m8.588802518s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.717831707 (597h35m8.588914752s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.743536652 (597h35m8.614619687s) [noise=true , address=145.79.227.43:4187 , piece_id=NQCXR5HGRYEQ3WW27PQJLMUOEEUXZDGGQJQQO5NYGHPXPAQFWG7A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.848591082 (597h35m8.719674127s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.899414969 (597h35m8.770498014s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.123129336 (597h35m8.994212371s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.211969381 (597h35m9.083052416s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.294556076 (597h35m9.165639142s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.479068837 (597h35m9.350151872s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.510445983 (597h35m9.381529018s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.536218769 (597h35m9.407301814s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.566859122 (597h35m9.437942167s) [noise=true , address=145.79.227.33:4877 , piece_id=BMMH4UJO7OQRDWK4QSSNCA3FVEUYXTX3TPWO62WMUG6DMQ5VTBEA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.619658661 (597h35m9.490741696s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.621672355 (597h35m9.49275539s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.621683898 (597h35m9.492766943s) [seg_pos={0 0}, upload=523818]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:04:09.621901051 (597h35m9.492984086s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: expected piece hash; context canceled; EOF
Dec 30 13:04:09.621910259 (597h35m9.492993294s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622063460 (597h35m9.493146505s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection
Dec 30 13:04:09.622087205 (597h35m9.49317025s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622135457 (597h35m9.493218492s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF
Dec 30 13:04:09.622142100 (597h35m9.493225135s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622242572 (597h35m9.493325607s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF
Dec 30 13:04:09.622247882 (597h35m9.493330927s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622337151 (597h35m9.493420186s) [seg_pos={0 0}, upload=523818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:04:09.622456739 (597h35m9.493539795s) [upload=523818]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:04:09.622466388 (597h35m9.493549433s) [upload=523818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:04:09.936998074 (597h35m9.808081119s) [upload=523818]: Done uploading segments...
Dec 30 13:04:09.937018172 (597h35m9.808101217s) [upload=523818]: Upload finished. err: <nil>
Duration: 31.23147835s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......X.......................X....X.......................] duration:
19.445552844s trace:
https://jaeger.dev.storj.io/trace/7b47a373389cd7c3
FAIL (2025-12-30 13:09:42 UTC)
orbiter warning: context deadline exceeded
Duration: 19.445578753s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X.XX..X..X.XX...............X..X.X..X....X........XX......] duration:
6.253258436s trace:
https://jaeger.dev.storj.io/trace/4a02b772b0018a96
FAIL (2025-12-30 13:09:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A) context deadline exceeded
======LOGS======
Dec 30 13:05:17.988231417 (597h36m17.859314452s) [upload=523833]: Starting upload
Dec 30 13:05:17.988290189 (597h36m17.859373224s) [upload=523833]: Done starting upload
Dec 30 13:05:17.988476343 (597h36m17.859559378s) [upload=523833]: Uploading segments...
Dec 30 13:05:17.988522001 (597h36m17.859605036s) [upload=523833]: Got next segment. Inline: false
Dec 30 13:05:17.988548350 (597h36m17.859631385s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 13:05:17.988552870 (597h36m17.859635925s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 13:05:18.235197488 (597h36m18.106280523s) [seg_pos={0 0}, upload=523833]: Begin upload segment...
Dec 30 13:05:18.235244077 (597h36m18.106327112s) [seg_pos={0 0}, upload=523833]: Done begin upload segment.
Dec 30 13:05:18.235285987 (597h36m18.106369022s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235320202 (597h36m18.106403247s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235847166 (597h36m18.106930201s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.235886951 (597h36m18.106969996s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236164309 (597h36m18.107247344s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236275291 (597h36m18.107358336s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236525807 (597h36m18.107608842s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236699468 (597h36m18.107782503s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.236797896 (597h36m18.107880931s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237062679 (597h36m18.108145714s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237424669 (597h36m18.108507704s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.237720462 (597h36m18.108803497s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238346965 (597h36m18.10943s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.238818193 (597h36m18.109901228s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239195892 (597h36m18.110278927s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:18.239419397 (597h36m18.110502442s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Uploading piece...
Dec 30 13:05:19.938524132 (597h36m19.809607167s) [upload=523833]: Next returned nil segment
Dec 30 13:05:19.938538439 (597h36m19.809621474s) [upload=523833]: Waiting for error group managing segments...
Dec 30 13:05:20.225416789 (597h36m20.096499834s) [noise=true , address=145.79.227.46:4602 , piece_id=YG7Y3JZELTLE2NM2LLDLRXB3MU2TQKXZGPTKWM5VNRJIZHHRO27Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.230259981 (597h36m20.101343026s) [noise=true , address=145.79.227.3:4151 , piece_id=4SPC56GTTPAUOEQS6WIYKT4XS5I7TH4NJTNL2SH44CUB4AGC7S5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.231961621 (597h36m20.103044666s) [noise=true , address=145.79.227.20:4438 , piece_id=G2LUF234AWZ5GIL6DTUIWVBGTPR6B6RZDVGKBSRULCNZJOXDVN5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.236052922 (597h36m20.107135967s) [noise=true , address=145.79.227.52:4447 , piece_id=4UTPOVLC4FIAGMAGULQDRCKKS4FDSADZ36A62VHQ7Y5YJKWKSNZQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.245330678 (597h36m20.116413723s) [noise=true , address=145.79.227.9:4366 , piece_id=3HO7WFYJ5V7QMA4ULDRZ2QUDFC3X3ARDR2JLL4SBEH3R3442LEEQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246275477 (597h36m20.117358512s) [noise=true , address=145.79.227.25:4617 , piece_id=3IPLSEQ4QUYBTD3J5RGBRVBB7MJ2BPJZ2SJWPLX2GULITVPHEZVQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.246765681 (597h36m20.117848726s) [noise=true , address=145.79.227.2:4115 , piece_id=Y3ZWB3BFPATJ3NUY7OUVPUO3KJ7QSURWXP6EGJY4D6CRLIYJSZ3Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.247080049 (597h36m20.118163084s) [noise=true , address=145.79.227.32:4834 , piece_id=6CVFHJDNVJXYCFQSKYJ4JGE45AFLLUZJBKG53GF7TGPOPUBG2YLQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.271725659 (597h36m20.142808704s) [noise=true , address=145.79.227.31:4799 , piece_id=X6UQZM7YVLMCR5X35EGQ7N6RBGYDYRJMI2YH7DV3CAORBDUMFHAQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.306214921 (597h36m20.177297956s) [noise=true , address=145.79.227.12:4767 , piece_id=I2U5XKOLG2WSPVXF7MGXXSGJQIFFSJ46LZLNLATI7EMGYWTACI6Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.311072280 (597h36m20.182155325s) [noise=true , address=145.79.227.50:4375 , piece_id=BYGKORNIBEJSJYBLZTWF2JHW4DC65I4O5WQG2AUU6AZBU2NVTEVA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.394910160 (597h36m20.265993206s) [noise=true , address=77.90.31.15:4697 , piece_id=KBZYZPT6HFCWY36ZQK37N6XQNZVFNOMLAGRAPJAA76MYQ2KBSX3A, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484099100 (597h36m20.355182136s) [noise=true , address=77.90.31.21:4877 , piece_id=OVD3QYJFN6RPTEPXFSZWSXF3J2YXCXZTUEWWYJJMYF4OZ4BRCZDQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: <nil>
Dec 30 13:05:20.484177910 (597h36m20.355260965s) [seg_pos={0 0}, upload=523833]: Segment reached optimal threshold of 13 pieces.
Dec 30 13:05:20.484451902 (597h36m20.355534947s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484461801 (597h36m20.355544836s) [noise=true , address=45.91.116.11:4117 , piece_id=CDYF2GSH5JXBJZ6U4MLTRZPKJV7JMZDOC7UXWRYLWVDLSYKRJK5Q, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484631704 (597h36m20.355714749s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 30 13:05:20.484643737 (597h36m20.355726772s) [noise=true , address=145.79.227.34:4905 , piece_id=J32TZMXC5FM3FPJM674QCXWHRLN2KQT2F6KYNCM3KA5GO4UY4CHA, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484847886 (597h36m20.355930931s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: context canceled; context canceled
Dec 30 13:05:20.484857505 (597h36m20.35594054s) [noise=true , address=77.90.31.13:4626 , piece_id=V27YSE6V3NJYGPHWOXJSJUNXNWXFPUX3TEOK4B7VXSFIA26Y5CPQ, seg_pos={0 0}, upload=523833]: Long tail cancellation due to optimal threshold
Dec 30 13:05:20.484881309 (597h36m20.355964355s) [seg_pos={0 0}, upload=523833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 13:05:20.484989906 (597h36m20.356072951s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:05:20.484997742 (597h36m20.356080777s) [upload=523833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:05:20.779558830 (597h36m20.650641885s) [upload=523833]: Done uploading segments...
Dec 30 13:05:20.779584489 (597h36m20.650667544s) [upload=523833]: Upload finished. err: <nil>
Duration: 6.253274017s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [...X.....X.X...X.........X....X........X..............X.....] duration:
31.124137983s trace:
https://jaeger.dev.storj.io/trace/59feccf4c7c3ba3a
FAIL (2025-12-30 13:09:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: quiescence
======LOGS======
ec 30 13:04:06.775086282 (597h35m6.646169327s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.775516872 (597h35m6.646599907s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.775911253 (597h35m6.646994298s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.776366340 (597h35m6.647449395s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.777866967 (597h35m6.648950012s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.778495334 (597h35m6.649578389s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.779041454 (597h35m6.650124499s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.779558509 (597h35m6.650641554s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.780053321 (597h35m6.651136366s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.780550918 (597h35m6.651633963s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.781069085 (597h35m6.65215213s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.781978246 (597h35m6.653061291s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.782926150 (597h35m6.654009195s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.783851333 (597h35m6.654934388s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.784805539 (597h35m6.655888584s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:06.786955854 (597h35m6.658038899s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Uploading piece...
Dec 30 13:04:08.223121485 (597h35m8.09420453s) [upload=523818]: Next returned nil segment
Dec 30 13:04:08.223139148 (597h35m8.094222193s) [upload=523818]: Waiting for error group managing segments...
Dec 30 13:04:08.717719473 (597h35m8.588802518s) [noise=true , address=145.79.227.5:4253 , piece_id=LOBQRRK5BLWDTAPUJB42YT3NC2G6M2J7EU3ALUMLTRD77NPYYFLQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.717831707 (597h35m8.588914752s) [noise=true , address=145.79.227.51:4413 , piece_id=IGKBEDGRF5ZTI7UUPASHF5IFZWJSJZ2V6ATPHWTNW3SRMDSPRCIA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.743536652 (597h35m8.614619687s) [noise=true , address=145.79.227.43:4187 , piece_id=NQCXR5HGRYEQ3WW27PQJLMUOEEUXZDGGQJQQO5NYGHPXPAQFWG7A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.848591082 (597h35m8.719674127s) [noise=true , address=77.90.31.15:4708 , piece_id=K6SXMEM4SWGSWASJT46GOND2LF4REXOJ2RZJCPCGGQU7JJ2VGKDA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:08.899414969 (597h35m8.770498014s) [noise=true , address=64.49.13.27:4964 , piece_id=RFCAZJ226BWPK37KCUOIJ7UQ2UJLDQQ6YILNZO67FEONKRA5LB6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.123129336 (597h35m8.994212371s) [noise=true , address=77.90.31.18:4836 , piece_id=NQ3OA633Y5TGCVGP5QCDHVG762ERUB5MB5GDLTCXIAX5S5XLJUQA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.211969381 (597h35m9.083052416s) [noise=true , address=45.91.116.8:4061 , piece_id=G5N5V7ISYQYRIPRSQI6YKO6HESVJOIF7N4GDYY7UJCPCUAZUER6A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.294556076 (597h35m9.165639142s) [noise=true , address=77.90.31.12:4597 , piece_id=YMNNNTOZWOQ2VYT4VWVW7VHIUCQKZ6GMQMAYSUEQAEISXYO4SKRA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.479068837 (597h35m9.350151872s) [noise=true , address=193.149.164.143:4703 , piece_id=WFBUJIEKQUXYFSSVB27M2YEEILDNN55E5KL23EUCUC2JDR3Z7XSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.510445983 (597h35m9.381529018s) [noise=true , address=145.79.227.44:4226 , piece_id=UPBKK6FMKBJTSNLDVHWP7BBVRMIHESW3ED7CQLUMEXRZGD7LYO5A, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.536218769 (597h35m9.407301814s) [noise=true , address=145.79.227.30:4793 , piece_id=VWYFBUFM4NKIK7G7UXVUYAFZT3YYUW42W65PKWE4QKF7L2IO4YYA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.566859122 (597h35m9.437942167s) [noise=true , address=145.79.227.33:4877 , piece_id=BMMH4UJO7OQRDWK4QSSNCA3FVEUYXTX3TPWO62WMUG6DMQ5VTBEA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.619658661 (597h35m9.490741696s) [noise=true , address=145.79.227.49:4344 , piece_id=Z6LDJHQ6KLS3O6HDUIJI5EAGKH3Y7ZYIKFTXSGRVTTBOTVUCSRSQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.621672355 (597h35m9.49275539s) [noise=true , address=45.91.116.14:4222 , piece_id=4JWMPKJRZPU2FDGXCS5JW5CP6OL3T3XLNGEUTXEGSDBBKHMUFIOA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: <nil>
Dec 30 13:04:09.621683898 (597h35m9.492766943s) [seg_pos={0 0}, upload=523818]: Segment reached optimal threshold of 14 pieces.
Dec 30 13:04:09.621901051 (597h35m9.492984086s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS): protocol: expected piece hash; context canceled; EOF
Dec 30 13:04:09.621910259 (597h35m9.492993294s) [noise=true , address=77.90.31.18:4823 , piece_id=CZMTHJYNM5JF6VOFS7FTNABHNIWJTUTPHFRYYR63E2NNIKKCTWEQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622063460 (597h35m9.493146505s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rfBAoVRdhYiRJFWqnamoRAFLxEUsk7tShJEG3jvhKDbcHwipD): protocol: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33424->45.91.116.9:4289: use of closed network connection
Dec 30 13:04:09.622087205 (597h35m9.49317025s) [noise=true , address=45.91.116.9:4289 , piece_id=PFVRYB6YFVSYFD4ZSS7CDLORUXSG3VGG2SE4ZNE5YBFQTRL5MEJQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622135457 (597h35m9.493218492s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF
Dec 30 13:04:09.622142100 (597h35m9.493225135s) [noise=true , address=145.79.227.40:4076 , piece_id=XGXIJP6AWHH322IXAUG3QEOO2AEPBEO6E52X6VGC226CAZUKQ7NA, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622242572 (597h35m9.493325607s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8): protocol: expected piece hash; context canceled; EOF
Dec 30 13:04:09.622247882 (597h35m9.493330927s) [noise=true , address=145.79.227.34:4936 , piece_id=K3HBCI7M3WQIJ3GMWMQN33F3UQPWUMQW33NT5H52BNKDD7QNCQRQ, seg_pos={0 0}, upload=523818]: Long tail cancellation due to optimal threshold
Dec 30 13:04:09.622337151 (597h35m9.493420186s) [seg_pos={0 0}, upload=523818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 13:04:09.622456739 (597h35m9.493539795s) [upload=523818]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 13:04:09.622466388 (597h35m9.493549433s) [upload=523818]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 13:04:09.936998074 (597h35m9.808081119s) [upload=523818]: Done uploading segments...
Dec 30 13:04:09.937018172 (597h35m9.808101217s) [upload=523818]: Upload finished. err: <nil>
Duration: 31.124169482s
core
uplink
us-select-1
us1-placement-20
|